loadADDEImage() not working for GOES-16 from geoarc server

Post any questions, ideas, or topics related to Jython and Python scripting.
Post Reply
User avatar
hproe
Posts: 503
Joined: Sat Nov 27, 2010 3:46 pm

loadADDEImage() not working for GOES-16 from geoarc server

Post by hproe »

Hi -

I note that loadADDEImage() appears not to work for GOES-16 geoarc server. It just issues an exception. Similar requests from other geoarc servers are working fine. Am I missing something? Can you replicate this?

HP
User avatar
bobc
Posts: 933
Joined: Mon Nov 15, 2010 5:57 pm

Re: loadADDEImage() not working for GOES-16 from geoarc server

Post by bobc »

Hi HP -

Here's a basic script that I'm able to run in today's nightly, as well as McV 1.6, that pulls in and displays data off geoarc with loadADDEImage:

Code: Select all

# Python dictionary
g16Parms = dict(
    server = 'geoarc.ssec.wisc.edu',
    dataset = 'AGOES16',
    descriptor = 'M1',
    day = '2017-09-05',
    time = '18:00:53',
    band = 10,
    unit = 'TEMP',
    size = 'ALL',
    )

# loadADDEImage call
g16Data = loadADDEImage(**g16Parms)

# Display the data returned from loadADDEImage in the currently active display panel
g16Layer = activeDisplay().createLayer('Image Display', g16Data)

Note that I didn't specify accounting in my g16Parms dictionary. It isn't necessary to specify accounting= with loadADDEImage if the dataset is defined in your remote ADDE Data Manager.

It is likely that you are doing something more complex than I did. Can you please post your script? If you don't want to post it to the forum, you can feel free to email it to the Help Desk. Also, please let me know the error that you are seeing.

Thanks -
Bob Carp
User avatar
hproe
Posts: 503
Joined: Sat Nov 27, 2010 3:46 pm

Re: loadADDEImage() not working for GOES-16 from geoarc server

Post by hproe »

Hi Bob -

Your sample code is running well. However, I am trying to retrieve full-disk imagery (image type = 'FD'). Adapting your code to such a retrieval I get a lengthy error message ending in
No images match given parameters (error code: -5000)


I have uploaded the method I am using to the SSEC server as 'genericImSequenceRemoteADDE.py'. You may test it with the following call

Code: Select all

sequence_sBand('2017-09-06 12:00UTC',1,900,'geoarc.ssec.wisc.edu','AGOES16','FD',14,'TEMP',(19.9,-66.7),(2500,3100),(1,1))

With me it issues '.... (partailly) missing frame'

Any clue?

HP
User avatar
bobc
Posts: 933
Joined: Mon Nov 15, 2010 5:57 pm

Re: loadADDEImage() not working for GOES-16 from geoarc server

Post by bobc »

Hi HP -

This GOES-16 ABI FD data actually goes down to the second. For example:

2017-09-06 20:00:35
2017-09-06 20:15:35
2017-09-06 20:30:35
2017-09-06 20:45:35

This means that you must specify the time down to the second in your loadADDEImage calls. Without specifying second, it assumes :00, or 0 seconds after the minute. With specifying seconds, I changed the following line in your function from:

Code: Select all

firstImageDT=DateTime.createDateTime(timeFirstImage,'yyyy-MM-dd HH:mmz')

to:

Code: Select all

firstImageDT=DateTime.createDateTime(timeFirstImage,'yyyy-MM-dd HH:mm:ssz')

And I changed your actual call of the function in the Jython Shell to:

Code: Select all

sequence_sBand('2017-09-06 12:00:35UTC',1,900,'geoarc.ssec.wisc.edu','AGOES16','FD',14,'TEMP',(19.9,-66.7),(2500,3100),(1,1))

... which works with the ":35" added to the time.

If this doesn't work for you, or if you have any other questions, please let me know.

Thanks -
Bob
User avatar
hproe
Posts: 503
Joined: Sat Nov 27, 2010 3:46 pm

Re: loadADDEImage() not working for GOES-16 from geoarc server

Post by hproe »

Hi Bob -

This is all well and works. So far, with the other GEO satellites, the seconds have not mattered. They were all =0 and methods like loadADDEImage were not bother when addressed with HH:mmz times strings. Basically I cannot see a problem ith having to state the seconds in additions. However, it appears that the number is not stable, e.g. for FD between 8 and 9 September it varies between 35 and 36 rather randomly:
08 00:00 to 03:30 35
08 03:45 to 23:30 36
08 23:45 to 05:45 35
09 06:00 to ??:?? 36
This gives me headaches when I want to extract an image sequence with my method. Any ideas on how to handle this issue?

HP
User avatar
bobc
Posts: 933
Joined: Mon Nov 15, 2010 5:57 pm

Re: loadADDEImage() not working for GOES-16 from geoarc server

Post by bobc »

Hi HP -

I added the following lines to your function before your "for loop":

Code: Select all

 # calculating second time to pass through loadADDEImage
 # this is necesary so a range can be used, since the user may
 # not enter an exact time of the image down to the second.  This
 # is done using timedelta, which is part of datetime, but not
 # VisAD.DateTime.  First, we create an actual datetime object
 # using timeFirstImage which is a variable passed through the
 # function, then apply a timedelta of 1 minute.
 from datetime import datetime as DT
 first_dt = DT.strptime(timeFirstImage, "%Y-%m-%d %H:%M:%SUTC")
 second_dt = first_dt +datetime.timedelta(seconds=60)
 # second_dt includes yyyy-mm-dd which we want to remove so we only
 # have hh:mm:ss.  To do this, I'll change the second_dt datetime
 # object to a string and remove the first 11 characters
 second_dt_str = str(second_dt)[11:]

And I modified your bandData=loadADDEImage line to:

Code: Select all

   bandData=loadADDEImage(day=dayString, 
      time=(timeString,second_dt_str),**addeParams)

... note that I changed the time= part to set the second time to second_dt_str

The majority of the code above are comments, but there's a summary:

  • loadADDEImage accepts a time range. To get things working, I decided that passing a time range of one minute though loadADDEImage would cover this case where the seconds weren't correct.
  • I found that you can create a datetime timedelta object (through some Google searching). To utilize this, I first created a datetime object from the timeFirstImage string that the user passes through the function by using datetime/strptime. I then used datetime.timedelta to add 60 seconds to this time, which I defined as second_dt.
  • At this point, second_dt includes yyyy-mm-dd, which can't be included in the time= keyword of loadADDEImage. I'm sure there are ways that you can re-format this datetime object to remove the yyyy-mm-dd, but I just converted it to a string and chopped off the first 11 characters and defined it as second_dt_str. After this, the string is in the format of "HH:mm:ss".
  • I utilized this second_dt_str object by including it in the loadADDEImage command, setting it as the second value passed through time=

With these changes, I'm able to run the following line from the Jython Shell. This would have errored before in the past, since the actual time of the image is 12:00:35, not 12:00:00:

Code: Select all

sequence_sBand('2017-09-06 12:00:00UTC',1,900,'geoarc.ssec.wisc.edu','AGOES16','FD',14,'TEMP',(19.9,-66.7),(2500,3100),(1,1))

If you want to, you could customize this further by adding a new keyword to your sequence_sBand() function where a user could include this timedelta value instead of hard-coding it to 60 seconds inside the function. loadADDEImage will search the server for all times in the dataset between the times passed through the function and return the most recent one. For example, if you ran a loadADDEImage call where you have time=('12:00:O00','14:00:00') pointing towards 1-minute GOES-16 mesoscale data, the 13:59:54 time is returned.

Another scenario I'm considering is if you specify a time very close to the changing of a day (for example, '23:59:40'). At this point, you would want to modify your loadADDEImage command further by setting day= to be a range as well. This is because the timedelta of one minute would change to a different date. For example, you could pull the day of the second_dt_str using:

Code: Select all

second_day = str(second_dt)[:11]

And then modify your loadADDEImage command to:

Code: Select all

   bandData=loadADDEImage(day=(dayString,second_day), 
      time=(timeString,second_dt_str),**addeParams)

If you give this a try and it doesn't work, or if you have any questions, please let me know.

Thanks -
Bob
User avatar
hproe
Posts: 503
Joined: Sat Nov 27, 2010 3:46 pm

Re: loadADDEImage() not working for GOES-16 from geoarc server

Post by hproe »

Hi Bob -

Thank you for the code proposal. Unfortunatley, this only works when retrieving a single frame. I think the delta time has to be constructed inside the loop, by introducing the 2 lines marked with hashtags:

Code: Select all

 for i in range(numImages):
  try:
   currentDT=DateTime(firstImageDT.getReal().getValue() +
      i*timeStep) # !!!native time step in [s] !!!
   deltaDT=currentDT+60     ###########
   timeZone=currentDT.getFormatTimeZone()
   dayString=currentDT.formattedString('yyyyDDD',timeZone)
   timeString=currentDT.formattedString('HH:mm:ss',timeZone)
   deltaString=deltaDT.formattedString('HH:mm:ss',timeZone)      ###########
   bandData=loadADDEImage(day=dayString,
      time=(timeString,timeString),**addeParams)
   print i+1,'   ',dayString,timeString
  except:
   print i+1,'   ',dayString,timeString,'  (partially) missing frame'
  else:
   imageSeq.append(bandData)
   

Note that I have not changed the 'bandData=...' yet, so with the exact time string given it should still work as before. However, it falls through to 'except'. When I outcomment the second line 'deltaString=...' it works again fine. I cannot see why this second line should influence the ADDE call. Can you help, please.

HP
User avatar
bobc
Posts: 933
Joined: Mon Nov 15, 2010 5:57 pm

Re: loadADDEImage() not working for GOES-16 from geoarc server

Post by bobc »

Hi HP -

Apologies on the delayed response, I was out of the office for a while. You are correct that the second time (referenced as deltaString) in your code would need to be calculated inside the loop so it works with all of the image times. I think that one of the issues is that deltaDT in your code is type visad.Real which doesn't have the ability to use the .formattedString method. currentDT is type visad.DateTime, which does have the .formattedString method.

I'm not entirely sure that this is the cleanest approach, but I'm getting things to work by modifying your code to:

Code: Select all

 for i in range(numImages):
  try:
   currentDT=DateTime(firstImageDT.getReal().getValue() +
      i*timeStep) # !!!native time step in[s] !!!
   
   # get the delta time and format string
   from datetime import datetime, timedelta
   deltaDT=currentDT+60
   t0 = datetime(1970, 1, 1)
   seconds = float(str(deltaDT))
   second_dt = t0 + timedelta(seconds=seconds)
   second_dt_str = str(second_dt)[11:]

   timeZone=currentDT.getFormatTimeZone()
   dayString=currentDT.formattedString('yyyyDDD',timeZone)
   timeString=currentDT.formattedString('HH:mm:ss',timeZone)
   bandData=loadADDEImage(day=dayString,
      time=(timeString,second_dt_str),**addeParams)
   print i+1,'   ',dayString,timeString
  except:
   print i+1,'   ',dayString,timeString,'  (partially) missing frame'
  else:
   imageSeq.append(bandData)

Note the blocked off part of code with the comment of "get the delta time and format string". Here's a brief summary of each line in the code block:
  1. Imports
  2. The same as your line, adding 60 seconds to currentDT. Note that this is a visad.Real type and can have a value like "1.50469926E9".
  3. The deltaDT in the above line is the number of seconds since epoch. This line defines epoch as 01/01/1970.
  4. For use with timedelta in the next line, seconds has to change to a float. float accepts strings, so deltaDT is set as a string.
  5. Adding deltaDT to the epoch time. This is a datetime object that returns something like "2017-09-06 12:01:00" when you print it.
  6. Format the second_dt_str to pull out just the timestamp.

If you give this a look, let me know if you have any problems/questions.

Thanks -
Bob
User avatar
hproe
Posts: 503
Joined: Sat Nov 27, 2010 3:46 pm

Re: loadADDEImage() not working for GOES-16 from geoarc server

Post by hproe »

Hi Bob -

As usual excellent advice. I got it working.

The FD image type I am usually using appears to be the only image type available from the server that has a variable image start time. What is the time range of these start times? If this range would be rather much smaller than 60s, one could think of shortening the interval to below 30s, in order to accommodate the other image types in the same method.

cheers, HP
User avatar
bobc
Posts: 933
Joined: Mon Nov 15, 2010 5:57 pm

Re: loadADDEImage() not working for GOES-16 from geoarc server

Post by bobc »

Hi HP -

Looking at all of the FD times off our archive server over the past 2 months, here's what I found:

  • With scan mode 3 (the normal one with 15 minute FD imagery), the times look to range from :33 to :45 seconds after the minute. There look to be several days in a row where it will hang in the range of :33 to :36 second after the minute. Then, for several days, it will hang around :40 to :43 seconds after the minute.
  • When the ABI goes to scan mode 4 with 5 minute FD imagery (e.g. on 08/20 and 04/21) the times are all within the range of :20 to :22 seconds after the minute.

As of right now, there's no way via ADDE to determine if an image is scan mode 3 or 4 short of looking at the time of the image, where the 15 minute FD times are :00, :15, :30, and :45 after the hour, and 5-minute imagery will be every 5 minutes. If you are loading the data via loadGrid then you could either use the filename, where it will include M3 or M4 for mode 3 or mode 4, or there is a global attribute contained in the files (timeline_id) that tells you the scan mode.

Hopefully this is enough guidance to help out with your function/script. If not, please let me know.

Thanks -
Bob
Post Reply