Finding multiple times in point data

[joleenf]

Hi,

I have loaded a point data file into the field selector, and when I try to display, I get an error message that I have more than one point with the same time in the time series. I can't find the problem in my original file, so I am trying to view the data as loaded into McIDAS-V. In the jython shell, I select the data, and have the following:

Code: Select all

a=selectData()
whatType(a[0])

FunctionType:
Domain has 3 components:
0. RealType: Latitude
0. Name = Latitude
0. Unit: deg
1. RealType: Longitude
1. Name = Longitude
1. Unit: deg
2. RealType: Altitude
2. Name = Altitude
2. Unit: m
Range:
RealTupleType has 2 components:
0. RealType: Util_RealType[unit:]
0. Name = Util_RealType[unit:]
0. Unit:
1. RealType: track_time_s_since_1970-01-01_00:00:00_000_UTC
1. Name = track_time_s_since_1970-01-01_00:00:00_000_UTC
1. Unit: s since 1970-01-01 00:00:00.000 UTC

Then, the times are in the flatField:
((Latitude, Longitude, Altitude) -> (Util_RealType[unit:], track_time_s_since_1970-01-01_00:00:00_000_UTC))

How do I select the track time so that I can use find?
[Make something like this work c=find(a[0], 'eq', java.util.Date('2014-02-12 14:00:00 UTC')]

Joleen

[mhiley]

Hi Joleen,

Would it be possible to post the data? I'm not sure about this one without being able to tinker...

-Mike

[joleenf]

Hi Mike,

ftp://ftp.ssec.wisc.edu/pub/ssec/joleenf/small.txt

Joleen

[mhiley]

Thanks,

I loaded the file in as "Text Point Data Files" from the "Files/Directories" chooser.

When the "selectData()" window pops up, I choose "Point Data".

I ran the following code to search for obs with matching location AND timestamp:

Code: Select all

a = selectData() # assume data source w/ data type "Text Point Data Files" from file "small.txt", choose "Point Data"

# loop through obs
for i in range(len(a)):
    ob = a[i]
    # search the remaining obs for duplicate earth locations
    for j in range(len(a)-i-1):
        ob2 = a[i+j+1]
        if ob.getEarthLocation().equals(ob2.getEarthLocation()):
            # if location is duplicate, check if timestamp matches
            if ob.getDateTime().equals(ob2.getDateTime()):
                # duplicate found, print them both to confirm
                print ob.getEarthLocation()
                print ob.getDateTime()
                print ob2.getEarthLocation()
                print ob2.getDateTime()

The results show one duplicate:

-10.76 53.71 1.0
2014-02-12 20:00:00 UTC
-10.76 53.71 1.0
2014-02-12 20:00:00 UTC

After a quick search in the txt file it looks like the culprits are lines 5323 and 5325.

Does that solve the problem? (It wouldn't be too hard to generalize this code to explicitly identify line numbers containing duplicates, if this is a common problem?...)
-Mike

[joleenf]

Hi Mike,

Duplicates should not be occurring in the file, the code which checks dates to determine if a particular image has been analyzed is not working. My jython code needs to be updated to use listADDEImages and run at exact times. Currently the code uses position number (pos=-1 and pos=-2).

I was loading the data using the wrong (less than ideal for the display type needed) chooser. I was using the text point and trajectory data files chooser, rather than text point data. Text point data is the cleaner option for this type of data.

If I did load the data via the text point and trajectory data files chooser, the JPythonMethod extract http://www.ssec.wisc.edu/visad-docs/javadoc/visad/python/JPythonMethods.html#extract(visad.Field,%20int), works to isolate the date/time values into a visad.FlatField.

Code: Select all

# assuming the data was loaded with "Text point and Trajectory Data File Chooser" so that a.getType() is (Time -> ((Latitude, Longitude, Altitude) -> (Util_RealType[unit:], track_time_s_since_1970-01-01_00:00:00_000_UTC)))

a=selectData()
fieldOfTimes=extract(a[0],'track_time_s_since_1970-01-01_00:00:00_000_UTC')

After that, I tried using find:

Code: Select all

for index in range(fieldOfTimes.getDomainSet().getLength()):
   singleTime=fieldOfTimes.getSample(index)
   matches=find(fieldOfTimes,'eq',singleTime)
   if len(matches) > 1:
      print matches

However, the "find" http://www.ssec.wisc.edu/visad-docs/javadoc/visad/python/JPythonMethods.html#find(visad.FieldImpl,%20java.lang.String,%20double) method is not matching anything in the flatField (fieldOfTimes).

At this point, I don't need to use extract, I can use your template to find the information I need.

Thanks!

Joleen

[mhiley]

Hey Joleen,

In your code above, shouldn't

Code: Select all

if len(matches) > 1:

be

Code: Select all

if len(matches) > 0:

?

-Mike

[joleenf]

Hi Mike,

If the entry is in the array once, I don't worry, if the entry is recorded twice or more, I do.

In this case, I never get more the len(matches)=0 even though I should be getting len(matches)>=1

Joleen