How to retrieve the file path+name using the formula editor?

How do I...?
Post Reply
User avatar
Diego_Souza
Posts: 6
Joined: Thu May 18, 2017 4:30 pm

How to retrieve the file path+name using the formula editor?

Post by Diego_Souza »

Hello all,

I'm calling a function using the formula editor. However, in the function, I need the file path and file name to do what I want (independent of the file a user chooses).

Here's the formula I'm using:
subsect_cmi(cmi_image[label=CMI Image to Subsect],min_lat[label=Min. Latitude,isuser=true,default=6.0],max_lat[label=Max. Latitude, isuser=true,default=24.0],min_lon[label=Min. Longitude, isuser=true,default=-95.0],max_lon[label=Max. Longitude,isuser=true,default=-61.0])

And here's my function:
def subsect_cmi(cmi_image,min_lat,max_lat,min_lon,max_lon):

min_lat = float(min_lat)
max_lat = float(max_lat)
min_lon = float(min_lon)
max_lon = float(max_lon)

# dictionary for loadGrid()
abiParms = dict(filename = cmi_image,field = "CMI",)
# loadGrid call for partial domain of data
partData = loadGrid(latLonBounds = (max_lat, min_lon, min_lat, max_lon), **abiParms)
# display the subsected image
partLayer = activeDisplay().createLayer("Image Display", partData)
# set enhancement for the new image
partLayer.setEnhancement("SVGAWVX_TEMP", range=(330, 161))
#return (cmi_image)


You may see that in the "dict" I need the filename. I tried to use the "cmi_image" passed by the function, but that's an image and not a string with the file name... so I'm getting an error. If I manually put the file name as a string, the function works fine...

Do you know how can I retrieve the file name and path from the cmi_image parameter, in order to use it inside the "dict"?

Thanks!
Diego
User avatar
bobc
Posts: 988
Joined: Mon Nov 15, 2010 5:57 pm

Re: How to retrieve the file path+name using the formula editor?

Post by bobc »

Hello -

You can add the following to your Jython Library:

Code: Select all

from javax.swing import JFileChooser
from ucar.unidata.util import FileManager

# Code to bring up an Open File window, allowing the user
# to select a local file, and the full path to the file will
# be returned as a string.
def getFileString():
    chooser = FileManager.MyFileChooser()
    result = chooser.showOpenDialog(None);
    if JFileChooser.APPROVE_OPTION == result:
        path = chooser.getSelectedFile().getPath()
        return str(path)

def subsect_cmi2(cmi_file,min_lat,max_lat,min_lon,max_lon):

    min_lat = float(min_lat)
    max_lat = float(max_lat)
    min_lon = float(min_lon)
    max_lon = float(max_lon)

    # dictionary for loadGrid()
    abiParms = dict(filename = cmi_file,field = "CMI",)
    # loadGrid call for partial domain of data
    partData = loadGrid(latLonBounds = (max_lat, min_lon, min_lat, max_lon), **abiParms)
   
    return partData

And then set up a formula to use these functions like:

Code: Select all

subsect_cmi2(getFileString(),min_lat[label=Min. Latitude,isuser=true,default=6.0],max_lat[label=Max. Latitude, isuser=true,default=24.0],min_lon[label=Min. Longitude, isuser=true,default=-95.0],max_lon[label=Max. Longitude,isuser=true,default=-61.0])

I set the description of the formula as: Subsect CMI
I set the name of the formula as: subsectCMI

To make use of the enhancement and range you specified in your script, you can set up a parameter default pointing to "subsectCMI" (the 'name' defined in the Formula Editor).

Once you select the "Subsected CMI" formula in the Fields panel of the Field Selector, you will first get the "Select input" window for the lat/lon values. After this, you will be presented with a "Open" dialog where you can select a local ABI file on your machine.

Please let me know if you have any questions about this.

Thanks -
Bob Carp
McIDAS Help Desk

P.S. Just as an FYI, if you post code in a forum post, I would suggest highlighting it with your mouse and then clicking the "Code" button at the top of the text field. This will allow your code to post like mine did above, where indentation is retained.
User avatar
Diego_Souza
Posts: 6
Joined: Thu May 18, 2017 4:30 pm

Re: How to retrieve the file path+name using the formula editor?

Post by Diego_Souza »

Hello Bob!

Thanks, I'll check this procedure. For now, here's what I'm doing:

Formula:

Code: Select all

subsect_cmi(file[type=file,isuser=true,label=CMI File to Subsect,filepattern=\.nc:netCDF Files],min_lat[label=Min. Latitude,isuser=true,default=6.0],max_lat[label=Max. Latitude, isuser=true,default=24.0],min_lon[label=Min. Longitude, isuser=true,default=-95.0],max_lon[label=Max. Longitude,isuser=true,default=-61.0],predefined[label=Predefined Region,isuser=true,type=choice,choices=None;South America;Central America;North America;Caribbean ])


Function:

Code: Select all

def subsect_cmi(cmi_image,min_lat,max_lat,min_lon,max_lon,predefined):

    # Turns off the Wireframe
    activeDisplay().setWireframe(False)

    # Select Region to Subsect According to the User Input
    if predefined == 'South America':
        min_lat = -60
        max_lat = 14
        min_lon = -20
        max_lon = -84
    elif predefined == 'Central America':
        min_lat = 6
        max_lat = 20
        min_lon = -75
        max_lon = -94
    elif predefined == 'North America':
        min_lat = 12
        max_lat = 55
        min_lon = -68
        max_lon = -120
    elif predefined == 'Caribbean':
        min_lat = 9
        max_lat = 28
        min_lon = -58
        max_lon = -87
    elif predefined == 'None':
        min_lat = float(min_lat)
        max_lat = float(max_lat)
        min_lon = float(min_lon)
        max_lon = float(max_lon)

    # Dictionary for loadGrid()
    abiParms = dict(filename = cmi_image,field = "CMI",)
   
    # LoadGrid call for partial domain of data
    partData = loadGrid(latLonBounds = (max_lat, min_lon, min_lat, max_lon), **abiParms)
   
    # Display the subsected image
    partLayer = activeDisplay().createLayer('Image Display', partData)
 
    # Set enhancement for the new image
    abi_band = (cmi_image[cmi_image.find("M3C" or "M4C")+3:cmi_image.find("_G16")])
   
    if (abi_band == '01' or abi_band == '02' or abi_band == '03' or abi_band == '04' or abi_band == '05' or abi_band == '06'):
        partLayer.setEnhancement("Gray Scale", range=(0,1))
    elif (abi_band == '07' or abi_band == '11' or abi_band == '12' or abi_band == '13' or abi_band == '14' or abi_band == '15' or abi_band == '16'):
        partLayer.setEnhancement("Longwave Infrared Deep Convection", range=(378,170))
    elif (abi_band == '08' or abi_band == '09' or abi_band == '10'):
        partLayer.setEnhancement("WVCOLOR35", range=(378,170))
   
    activeDisplay().setCenter(((max_lat+min_lat)/2),((max_lon+min_lon)/2),scale=1)
   
    partLayer.setColorScale(visible=True,placement='Left',showUnit=True)


The attachment "Input_Window" shows the formula input window.

The attachment "Result" shows the result for channel 13.

The attachment "Message" shows a message I get everytime I open an image.

I understand that the message is showing because I'm not returning an image in the end of the function... However, if I return the PartData, it is not configured with the color enhancement, and if I use the "Parameter Default" as you suggested, I can't use multiple colortables, for the function to apply one depending on the band I open...

The formula/function is working fine for my purpose, however I don't have any idea of how to get rid of that message.

Another issue I'm having is that when I open multiple images, a time sequence is created in the upper part of the window, however, when I click at the "Play" button, the blue box changes it position, but the image doesn't animate ("Animation Error" attachment)...

Regards,
Diego
Attachments
Message.png
Message.png (4.25 KiB) Viewed 4961 times
Animation_Error.png
Result.png
Input_Window.png
Input_Window.png (11.61 KiB) Viewed 4963 times
User avatar
bobc
Posts: 988
Joined: Mon Nov 15, 2010 5:57 pm

Re: How to retrieve the file path+name using the formula editor?

Post by bobc »

Hello -

I just wanted to let you know that I'm looking into this. It is an interesting problem and I understand why the parameter default won't work in your situation. One thing that I noticed is you get the "Selected image(s) not available" message if you choose the Image Display display type, but not "Color-Shaded Plan View". I've been trying to think of alternatives to get this working w/o error with the Image Display but I don't have anything yet. One thought was having two separate functions, one to display the original data and another to apply the enhancement based on the band, but I need to investigate this further.

Thanks -
Bob
User avatar
Diego Souza
Posts: 2
Joined: Thu May 18, 2017 4:27 pm

Re: How to retrieve the file path+name using the formula editor?

Post by Diego Souza »

Hi Bob!

I have tried to use the "Color-Shaded Plan View" but is much slower...

Thank you for the attention on this subject!

By the way, I visited the SSEC building / rooftop today with Mr. Allen Huang. Do you also work there? I travelled from Brazil to attend the CSPP / IMAPP users' meeting.

I'm leaving tomorrow (Jun 30th) by night.

What an awesome place!

Kind Regards,
Diego
User avatar
bobc
Posts: 988
Joined: Mon Nov 15, 2010 5:57 pm

Re: How to retrieve the file path+name using the formula editor?

Post by bobc »

Hello Diego -

Yes, that is the building I work in. I'm glad you enjoyed Madison!

I spent some more time investigating this and I spoke with a programmer. To get this working with the Image Display display type without the "Selected image(s) not available" message will actually require some coding changes in McIDAS-V, and it can't be resolved in the function itself or through the current Formula Editor dialog. As you mentioned earlier, we aren't actually returning any type of a data object from the function, and it's basically just a script wrapped up in a function, which McIDAS-V isn't necessarily designed to work with. Our programmer has an idea of a way to get this working. I'll speak with him again next week and I'll likely write up an inquiry so we can prioritize the work. When I get this written, I'll let you know the inquiry number so you can track the progress. I'll also let you know of any updates.

Thanks -
Bob
User avatar
bobc
Posts: 988
Joined: Mon Nov 15, 2010 5:57 pm

Re: How to retrieve the file path+name using the formula editor?

Post by bobc »

Hi Diego -

As an FYI, I wrote this up as Inquiry 2569. I made a note in the inquiry to contact you if there is any progress, but I can't make any promises on when this inquiry will be addressed. If you have any followup questions, please let me know.

Thanks -
Bob
User avatar
bobc
Posts: 988
Joined: Mon Nov 15, 2010 5:57 pm

Re: How to retrieve the file path+name using the formula editor?

Post by bobc »

Hi Diego -

It was great meeting you at the NOAA Satellite Conference. As for getting loops from these local ABI files, the issue is that the files don't have a "time" dimension included with them, as seen if you do a ncdump on the files. The way we are currently getting the time information (in the Time Animation Widget) from the data is through the "time_coverage_start" global attributes included with the data. Generally, McIDAS-V relies on having a "time" dimension available for aggregating all of the individual files into one data source so you can create a loop that plays through the Time Animation Widget. This is how the "Aggregate Grids By Time" Data Type in the General>Files/Directories chooser works. If you have a bunch of individual netCDF/grib files that have a "time" dimension, this data type aggregates all of the individual files into one data source that you can create a loop from. We have an inquiry written to add a similar data type that will allow for aggregating these ABI netCDF files together into one data source, but this work hasn't been completed.

The way I'm currently aggregating these files together is by creating a NcML wrapper file that extracts the date/time information from the file name. This is done using dateFormatMark, which you can read more about at that link. Here's how I have things set up right now:

  • I have a directory on my machine called C:/Users/rcarp/Data/ABI/FD.
  • Inside this directory, I have 16 directories for each band (e.g. "Band_1", "Band_2", ... "Band_16". Each directory contains 4 time-consecutive full disk files, such as "OR_ABI-L1b-RadF-M3C01_G16_s20171871500384_e20171871511150_c20171871511197.nc"
  • I also have a directory called "Wrappers", where I have NcML wrapper files for each band (e.g. "Band_1.ncml", "Band_2.ncml", ... "Band_16.ncml").

All of these NcML files are the same except for the "scan location". These files create one data source of EVERY file included in the scan location. Here's how my Band_14.ncml file looks:

Code: Select all

<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
  <aggregation dimName="time" type="joinNew">
    <variableAgg name="Rad"/>
    <variableAgg name="DQF"/>
    <remove name="t" type="variable"/>
    <scan location="../Band_14" dateFormatMark="OR_ABI-L1b-RadF-M3C14_G16_s#yyyydddHHmmss" suffix=".nc" subdirs="false"/>
      <attribute type="String" name="coordinates" value="band_id band_wavelength time y x"/>
  </aggregation>
</netcdf>

To get this NcML file working through a formula, I had to modify the function in order to loop through all of the timesteps included in the scan location directory. Here's my version of the function now. I added some in-line comments, but please let me know if any of it doesn't make sense:

Code: Select all

def subsect_cmi(cmi_image,min_lat,max_lat,min_lon,max_lon,predefined):

    # Turns off the Wireframe
    activeDisplay().setWireframe(False)

    # Select Region to Subsect According to the User Input
    if predefined == 'South America':
        min_lat = -60
        max_lat = 14
        min_lon = -20
        max_lon = -84
    elif predefined == 'Central America':
        min_lat = 6
        max_lat = 20
        min_lon = -75
        max_lon = -94
    elif predefined == 'North America':
        min_lat = 12
        max_lat = 55
        min_lon = -68
        max_lon = -120
    elif predefined == 'Caribbean':
        min_lat = 9
        max_lat = 28
        min_lon = -58
        max_lon = -87
    elif predefined == 'None':
        min_lat = float(min_lat)
        max_lat = float(max_lat)
        min_lon = float(min_lon)
        max_lon = float(max_lon)

    # Since we are working with multiple timesteps, return a list of date/time
    # information that can be passed through loadGrid
    abiTimes = listGridTimesInField(filename = cmi_image,field = "Rad")

    # Set up an empty list.
    abiList = []
    # Loop through all of the times returned from listGridTimesInField
    # Run loadGrid on each timestep, and append to the abiList created above
    for x in abiTimes:
        addeDict = dict(
            filename = cmi_image,
            field = "Rad",
            # The date/time information in each item of the list is in a format
            # like "2017-07-06T15:00:38Z".  We need to remove this "T" character
            # from each string in the list of dates/times and replace it with a
            # space, which is what ".replace" does below
            time = str(x).replace("T", " "),
            latLonBounds = (max_lat, min_lon, min_lat, max_lon),
        )
        partData = loadGrid(**addeDict)
        abiList.append(partData)

    # return abiList.  McIDAS-V can't accept a Python list as a result of a function
    # when calling the function through a formula.  makeFlatFieldSequence turns each
    # element of the list to a VisAD flat field which allows McIDAS-V to create a loop
    return makeFlatFieldSequence(abiList)

Now, when I evaluate the formula that calls this function, I can select my NcML file, and a loop is displayed. This still does not address modifying the enhancement after the data is plotted. That will likely have to be handled as part of Inquiry 2569 linked in my last response.

Let me know if you have any questions about this.

Thanks -
Bob
Post Reply