Shift GLM point data

How do I...?
Post Reply
User avatar
Ice Age Mark
Posts: 4
Joined: Thu Jun 01, 2017 9:27 pm

Shift GLM point data

Post by Ice Age Mark »

Hello,

I'm trying to use the script found here:

viewtopic.php?f=14&t=1663

to shift GLM data by a user specified Lat/Lon offset. Everything seems right
but I get this error no matter what:

Traceback (most recent call last):
File "<string", line 1, in <module
File "C:\Program
Files\McIDAS-V-System\lib\mcidasv-1.8-SNAPSHOT.jar\edu\wisc\ssec\mcidasv\resources\python\utilities\decorators.py",
line 97, in wrapper
File "<string", line 14, in nc_latlonshift
AttributeError: 'ucar.unidata.data.point.PointObTuple' object has no attribute
'getDomainSet'

I don't know what this means or how to correct it. The script seems to work for
others on the forum. I think I have everything set up correctly. A dialog opens for the Lat/Lon input, then a choice for the data shows. Then after choosing a GLM file, I get the error above.

Any help or insight would be greatly appreciated.

Mark
User avatar
bobc
Posts: 937
Joined: Mon Nov 15, 2010 5:57 pm

Re: Shift GLM point data

Post by bobc »

Hi Mark,

Thanks for posting your question to the forum. I was unable to get a function working that would create and display the shifted data as a point netCDF data source. Adding the ability to remove the parallax correction from GLM data is part of inquiry 2607. I'll add a note to the inquiry that you requested this.

I did come up with an alternative way of doing this, though, where you can write out the GLM points to a CSV file shifted lat/lon values. If you want to give it a try, here are the steps:

  1. Through the Jython Library, add the following function:

    Code: Select all

    def glmShiftCSV(data, latshift, lonshift, fileName):
        import csv
        timeList = []
        latList  = []
        lonList  = []
        for x in range(0, len(data)):
            latList.append(data[x].earthLocation[0] + float(latshift))
            lonList.append(data[x].earthLocation[1] + float(lonshift))
            timeList.append(str(data[x].dateTime).replace(' ','_'))
        header = ['Date/Time', 'Latitude', 'Longitude']
        with open(fileName, 'wb') as csv_out:
            mywriter = csv.writer(csv_out)
            mywriter.writerow(header)
            rows = zip(timeList, latList, lonList)
            mywriter.writerows(rows)
        return data

    Essentially, this function gets the time, lat, and lon of each point of data and adds a user-defined shift (in degrees) to the latitude and longitude values. fileName is a variable that'll be called in the formula, which is the full path and name of the CSV file to write your points to.
  2. Create a new formula to access this function. For example:
    Description: GLM to CSV
    Name: glmtocsv
    Formula: glmShiftCSV(data, latshift[isuser=true,default=1], lonshift[isuser=true,default=1], fileName[isuser=true,default=C:/Users/rcarp/glm_shift_1.csv])
    ... note that you'll want to change the fileName default to some path on your machine
    In the Settings tab, I chose "Use selected" for Displays and selected only "Point Data". Click OK Add Formula.
  3. Load your GLM data source into the Field Selector. Through the properties of the layer, modify the time binning as desired. You can also optionally go to the Spatial Subset tab of the Properties window and draw a bounding box around your area of interest. While not required, if you subset a relatively small region the display and creation of the CSV file will be faster, especially with a lot of time steps.
  4. Under Formulas in the Field Selector, choose the "GLM to CSV" formula and click Create Display.
  5. In the Select input window, enter the lat/lon shift values (in degrees, decimals work) and the file name and path. I left my path and file name in there as an example. Click OK.
  6. In the Field Selector window that appears, choose "Point Data" and click OK. This will plot the data points in their original locations and also write the CSV file.
  7. Back in the General>Files/Directories chooser of the Data Explorer window, navigate to the CSV file that was written in the previous step and select it. With the "Text Point Data Files" Data Type selected, click Add Source.
  8. In the Point Data window that appears, you'll have to tell McIDAS-V what the columns are. This is something that you'll only have to do the first time. At the bottom of the window, enter the following:

    Code: Select all

    Value:          Name:          Unit/Data Format:
    Date/Time:      Time           yyyy-MM-dd_HH:mm:SS
    Latitude:       Latitude       degrees
    Longitude:      Longitude      degrees

    Once you have all of this entered, you can click the Preferences button and choose "Save Current". Enter a name in the window that appears (such as "glm") and click OK. Click OK in the Point Data window. Note that next time you load one of these CSV files, you can click Preferences in the Point Data window, select "glm", and all of the values that you had to enter the first time for Name and Unit/Data Format will be automatically populated.
  9. From the Field Selector, choose the Point Data field, and click Create Display. This should display GLM points shifted at your defined values.

Please let me know if you give this a try and run into any problems.

Thanks,
Bob Carp
McIDAS User Services
User avatar
Ice Age Mark
Posts: 4
Joined: Thu Jun 01, 2017 9:27 pm

Re: Shift GLM point data

Post by Ice Age Mark »

Hello Bob,

Thanks for this. I was able to get it working correctly by changing the "SS" in the Preferences to "ss". I also needed the flash_energy value because I have my layout model set up to display the "+" scaled and colored by that field. So now the code looks like this (for others):

Code: Select all

def glm_shift(data, latshift, lonshift, fileName):
    import csv
    timeList = []
    latList  = []
    lonList  = []
    eList = []
    for x in range(0, len(data)):
        latList.append(data[x].earthLocation[0] + float(latshift))
        lonList.append(data[x].earthLocation[1] + float(lonshift))
        timeList.append(str(data[x].dateTime).replace(' ','_'))
        eList.append(data[x].getData()[1])
    header = ['Date/Time', 'Latitude', 'Longitude', 'flash_energy']
    with open(fileName, 'wb') as csv_out:
        mywriter = csv.writer(csv_out)
        mywriter.writerow(header)
        rows = zip(timeList, latList, lonList, eList)
        mywriter.writerows(rows)
    return data

The point data field names are now:

Code: Select all

Value:          Name:          Unit/Data Format:
Date/Time:      Time           yyyy-MM-dd_HH:mm:ss
Latitude:       Latitude       degrees
Longitude:      Longitude      degrees
flash_energy:      flash_energy      Joules


And the formula is:

glm_shift(data, latshift[isuser=true,default=1], lonshift[isuser=true,default=1], fileName[isuser=true,default=C:/Users/???/????/glm_shift_1.csv])

Many thanks,

Mark
User avatar
Ice Age Mark
Posts: 4
Joined: Thu Jun 01, 2017 9:27 pm

Re: Shift GLM point data

Post by Ice Age Mark »

Hello again,

I'm trying to automate this tedious process now and learn scripting in the process. I've set up what seems right, but I can't get csv writer to produce the exact two header lines required to re-load the csv file back in by script. It seems like I need opposite things from the csv writer for each line; no quotes for the 1st line, and quotes retained in the second. How can I get my script to produce this exact header for the exported csv file?:

(index)->(Time,Latitude,Longitude,flash_energy)
Time[fmt="yyyy-MM-dd_HH:mm:ss"],Latitude[unit="degrees"],Longitude[unit="degrees"],flash_energy[unit="Joules"]

Here is the script I'm using:

Code: Select all

def glm_shift(data, latshift, lonshift, fileName):
    import csv
    timeList = []
    latList  = []
    lonList  = []
    eList = []
    for x in range(0, len(data)):
        latList.append(data[x].earthLocation[0] + float(latshift))
        lonList.append(data[x].earthLocation[1] + float(lonshift))
        timeList.append(str(data[x].dateTime).replace(' ','_'))
        eList.append(data[x].getData()[1])
    header1 = '(index)->(Time,Latitude,Longitude,flash_energy)'
    header2 = 'Time[fmt="yyyy-MM-dd_HH:mm:ss"],Latitude[unit="degrees"],Longitude[unit="degrees"],flash_energy[unit="Joules"]'
    with open(fileName, 'wb') as csv_out:
        mywriter = csv.writer(csv_out)
        mywriter.writerow([header1])
        mywriter.writerow([header2])
        rows = zip(timeList, latList, lonList, eList)
        mywriter.writerows(rows)
    return data

The header I get is:

"(index)->(Time,Latitude,Longitude,flash_energy)"
"Time[fmt=""yyyy-MM-dd_HH:mm:ss""],Latitude[unit=""degrees""],Longitude[unit=""degrees""],flash_energy[unit=""Joules""]"

Using quoting=csv.QUOTE_NONE just sends me chasing the schmoo. Strip and replace don't work because it's a list object.

Any help getting this to work so I can continue learning/practicing scripting would be greatly appreciated.

Best regards,

Mark
User avatar
bobc
Posts: 937
Joined: Mon Nov 15, 2010 5:57 pm

Re: Shift GLM point data

Post by bobc »

Hi Mark,

Could you give this function a try?

Code: Select all

def glm_shift(data, latshift, lonshift, fileName):
    import csv
    timeList = []
    latList  = []
    lonList  = []
    eList = []
    for x in range(0, len(data)):
        latList.append(data[x].earthLocation[0] + float(latshift))
        lonList.append(data[x].earthLocation[1] + float(lonshift))
        timeList.append(str(data[x].dateTime).replace(' ','_'))
        eList.append(data[x].getData()[1])
    with open(fileName, 'wb') as csv_out:
        mywriter = csv.writer(csv_out, delimiter=",", quotechar="'", quoting=csv.QUOTE_NONE, escapechar=' ')
        mywriter.writerow(['(index)->(Time,Latitude,Longitude,flash_energy)'])
        mywriter.writerow(['Time[fmt="yyyy-MM-dd_HH:mm:ss"]', 'Latitude[unit="degrees"]', 'Longitude[unit="degrees"]', 'flash_energy[unit="Joules"]'])
        rows = zip(timeList, latList, lonList, eList)
        mywriter.writerows(rows)
    return data

The key here is in the "mywriter - csv.writer" line, where I specified delimiter, quotechar, quoting, and escapechar. I found that this combination seemed to work for me through trial and error, but I relied pretty heavily on the CSV File Reading and Writing page of the Python docs, specifically the Module Contents section. I was able to take the file written by the formula and load it right into McIDAS-V using the "Text Point Data files" data type in the General>Files/Directories chooser, meaning the header was written in a way that McIDAS-V could understand it. Here's how the header was written to the file using the modified function above:

Code: Select all

(index)->(Time ,Latitude ,Longitude ,flash_energy)
Time[fmt="yyyy-MM-dd_HH:mm:ss"],Latitude[unit="degrees"],Longitude[unit="degrees"],flash_energy[unit="Joules"]

Please let me know if you give this a try and run into any problems.

Thanks,
Bob
User avatar
Ice Age Mark
Posts: 4
Joined: Thu Jun 01, 2017 9:27 pm

Re: Shift GLM point data

Post by Ice Age Mark »

Hello there Bob,

Thank you so very much for figuring this out. I realize it's a Python question, not a McIDAS one, and I appreciate your indulgence. The csv writer documentation is already at the top of my Python resource links, but I just couldn't deduce the right combination of characters from it. It's reassuring that you also had to resort to trial and error, which I spent many hours doing as well; along with reading dozens of posts/blogs about such things.

The header still has some extra spaces in the first line and I don't know if that matters because:

I want to reload the csv file with a script, not manually, and I am having trouble with that too (see below). The csv already loaded manually without the special header, because I saved the load parameters as you suggested originally. I think I need that correct special header for a script to load the file.

Here's the bigger picture. Using the lat/lon shift script you so kindly created works great, but it's a tedious manual process (plus I'm old and have neurological problems). I want to learn some scripting in McIDAS so I imagined I would automate this and learn at the same time. I thought I would start with 3 simple scripts, one to download 10 minutes of GLM data, the one you created to extract a shifted csv file, and one to reload the csv and set things like declutter and layout model; and then combine them.

So, now I have my csv file with it's proper header (maybe?) like I read about in the documentation, and I want to reload it by script. I read that using

Code: Select all

load(file or URL)
would "find the correct TextAdapter based on file extension or structure" (https://www.ssec.wisc.edu/~billh/README.text) and my file resembles the form of example #8. Using the code below, the file seems to load (meaning it's present in the legend) the file (and has my declutter setting and layout model); but no points or time steps are displayed; so I'm missing something here.

Code: Select all

def load_csv():
    from visad.python.JPythonMethods import *
    #data = makeDataSource('C:\\Users\\????\\????\\glm_shift_1.csv')##This works, but I don't need it.
    data = load('C:\\Users\\????\\????\\glm_shift_1.csv')##This doesn't?
    layer=activeDisplay().createLayer('Point Data Plot',data)
    layer.setDeclutter(value=False)
    layer.setLayoutModel(model='Flash')
    return layer


Regarding step 1, downloading GLM files by script: It's seeming like it's not possible to download e.g the latest 30 files (10 min.) of GLM data from the THREDDS catalog source using a script. Is this correct? I don't see a way around the "Content-Length not supported" error I get. I was hoping that index numbers might work somehow since it's a file list, but I'm lost for any way to do this.

Any help or insight would be appreciated, the documentation seems very scant and opaque to a beginner. Yes, I know I can set up Favorites to do some of this, but I wouldn't learn anything new. (Hard learning is good learning!)

Many thanks again,

Mark
User avatar
bobc
Posts: 937
Joined: Mon Nov 15, 2010 5:57 pm

Re: Shift GLM point data

Post by bobc »

Hello,

Apologies for the delayed response. I spent some time looking into this and didn't get much further than you did. McIDAS-V has some scripting functions for loading in certain types of data, for example:

loadADDEImage - loads satellite data from local or remote servers
loadGrid - loads gridded data files (local or remote on a catalog)
loadJPSSImage - loads local JPSS (SNPP/NOAA 20) imagery

We don't currently have functions for loading CSV or local GLM data. I wrote up individual inquiries (#2194 - CSV and #2195 - GLM) to evaluate the development of these functions.

Since you're interested in GLM data, I figured I'd mention that the nightly build (1.9beta1) has an under development chooser that allows you to access and display remote GLM data from a server. You can't apply a parallax correction to the data as of right now (inquiry written for this development), but it's easier to do the binning. If you're interested, you can download the nightly here. The username and password for the download are both: mcv

I'd suggest downloading the installer for your platform that doesn't include java11 in the name. For example:

Windows: McIDAS-V_1.9beta1_windows64_installer.exe
Linux: McIDAS-V_1.9beta1_linux_x86-64_installer.sh
macOS: McIDAS-V_1.9beta1_macosx_x86_installer.dmg

Note that this nightly is automatically built every day with all of the previous day's programming changes. Because of this, not everything has been tested and there may be bugs. Here are some steps for working with remote GLM data through ADDE in the nightly:

  1. In the Data Sources tab of the Data Explorer, navigate to the Under Development > GLM chooser.
  2. Connect to adde.ucar.edu/RTGOESR and choose your GLM type. For example, choose "GOES-East Geostationary Lightning Mapper Flashes"
  3. In the Times panel, the Increment and Relative tabs are used together and work to bin the data together. For example, if you want 5 timesteps of 2-minute intervals (the last 10 minutes of data), you could set Increment to "2.0" and select "5 most recent" in the Relative Times tab. Click Add Source.
  4. In the Fields panel of the Field Selector, choose "GLM Lightning Data". Keep "Point Data Plot" selected in the Displays panel. Optionally, in the Region subset tab you could draw a bounding box around an area of interest. The smaller the domain, the faster things will go.

From here the display will look similar to what you end up with when using netCDF files through the General>Files/Directories or Catalogs choosers. You'll likely end up with 1 more time step than requested because of how the time rounding works. With this rounding, you'll see that either the first or last frame in the animation will include fewer points than the other frames. Again, there's no way to easily remove the parallax correction applied to the data, but at least this should be an easier way of plotting the original data.

Thanks,
Bob
Post Reply