VIIRS metadata retrieval

Post ideas for new functionality you'd like to see in McIDAS-V or ideas for new tutorials.
Post Reply
User avatar
hproe
Posts: 504
Joined: Sat Nov 27, 2010 3:46 pm

VIIRS metadata retrieval

Post by hproe »

Hi -

The following refers to DNB data from VIIRS, but might apply also to the other bands.

The method

Code: Select all

listVIIRSFieldsInFile(filename)
lists the metadata fields. However, the content of the list differs between data ordered with the geolocation package aggregated (GDBNO-SVDNB_npp...) or de-aggregated (SVDNB_npp...). In the aggregated case the list appears to be more complete than in the de-aggregated case. Also the Field Selector of the Data Explorer shows the same differences. Now I have two request.

1. The field list should be the same in both cases. The list should present most of if not all the fields given in the geolocation package. In particular, I need lunar/solar azimuth and zenith angles for contrast adjustment.

2. There is the method

Code: Select all

listVIIRSTimesInField(filename, field=none)
to retrieve date and time. It would be convenient to have similar methods for retrieving other metadata. In particular, I am interested in those given under point 1.

HP
User avatar
jayh
Posts: 424
Joined: Thu Jan 15, 2009 10:34 pm

Re: VIIRS metadata retrieval

Post by jayh »

Hi HP-

I wrote up inquiry 2475 for the expanding of the jython methods for VIIRS data interrogation.

I was testing with some DNB files and SVM files with and without the geolocation data combined in the files. I also downloaded 'deaggregated' files from CLASS which have each ~85 second granule as separate files. (I used 1.6 and the latest nightly.)

1. I did not see any differences in the Field Selector field listings among any of the data. All combinations of loading the data would list all the available variables. If you have some sample data files that show this behavior, I could try them here.

2. In the Jython shell I did see the 'incomplete' listing when using listVIIRSFieldsInFile(filename) with files that did not have the geolocation data combined. However, if I specified the individual geolocation file in the method, then I would get the missing variables listed. For example:

listVIIRSFieldsInFile('/Users/jayh/Data/joleen/SVM03_npp_d20140403_t1809429_e1811071_b12604_c20140404004022598570_noaa_ops.h5')
VIIRS-M3-SDR_All/ModeGran
VIIRS-M3-SDR_All/ModeScan
VIIRS-M3-SDR_All/NumberOfBadChecksums
VIIRS-M3-SDR_All/NumberOfDiscardedPkts
VIIRS-M3-SDR_All/NumberOfMissingPkts
VIIRS-M3-SDR_All/NumberOfScans
VIIRS-M3-SDR_All/PadByte1
VIIRS-M3-SDR_All/QF1_VIIRSMBANDSDR
VIIRS-M3-SDR_All/QF2_SCAN_SDR
VIIRS-M3-SDR_All/QF3_SCAN_RDR
VIIRS-M3-SDR_All/QF4_SCAN_SDR
VIIRS-M3-SDR_All/QF5_GRAN_BADDETECTOR
VIIRS-M3-SDR_All/Radiance
VIIRS-M3-SDR_All/Reflectance
VIIRS-M3-SDR_All/ReflectanceFactors
Data_Products/VIIRS-M3-SDR/VIIRS-M3-SDR_Aggr
Data_Products/VIIRS-M3-SDR/VIIRS-M3-SDR_Gran_0
[u'VIIRS-M3-SDR_All/ModeGran', u'VIIRS-M3-SDR_All/ModeScan', u'VIIRS-M3-SDR_All/NumberOfBadChecksums', u'VIIRS-M3-SDR_All/NumberOfDiscardedPkts', u'VIIRS-M3-SDR_All/NumberOfMissingPkts', u'VIIRS-M3-SDR_All/NumberOfScans', u'VIIRS-M3-SDR_All/PadByte1', u'VIIRS-M3-SDR_All/QF1_VIIRSMBANDSDR', u'VIIRS-M3-SDR_All/QF2_SCAN_SDR', u'VIIRS-M3-SDR_All/QF3_SCAN_RDR', u'VIIRS-M3-SDR_All/QF4_SCAN_SDR', u'VIIRS-M3-SDR_All/QF5_GRAN_BADDETECTOR', u'VIIRS-M3-SDR_All/Radiance', u'VIIRS-M3-SDR_All/Reflectance', u'VIIRS-M3-SDR_All/ReflectanceFactors', u'Data_Products/VIIRS-M3-SDR/VIIRS-M3-SDR_Aggr', u'Data_Products/VIIRS-M3-SDR/VIIRS-M3-SDR_Gran_0']


listVIIRSFieldsInFile('/Users/jayh/Data/joleen/GMTCO_npp_d20140403_t1809429_e1811071_b12604_c20140404003429456303_noaa_ops.h5')
VIIRS-MOD-GEO-TC_All/Height
VIIRS-MOD-GEO-TC_All/Latitude
VIIRS-MOD-GEO-TC_All/Longitude
VIIRS-MOD-GEO-TC_All/MidTime
VIIRS-MOD-GEO-TC_All/ModeGran
VIIRS-MOD-GEO-TC_All/ModeScan
VIIRS-MOD-GEO-TC_All/NumberOfScans
VIIRS-MOD-GEO-TC_All/PadByte1
VIIRS-MOD-GEO-TC_All/QF1_SCAN_VIIRSSDRGEO
VIIRS-MOD-GEO-TC_All/QF2_SCAN_VIIRSSDRGEO
VIIRS-MOD-GEO-TC_All/QF2_VIIRSSDRGEO
VIIRS-MOD-GEO-TC_All/SCAttitude
VIIRS-MOD-GEO-TC_All/SCPosition
VIIRS-MOD-GEO-TC_All/SCSolarAzimuthAngle
VIIRS-MOD-GEO-TC_All/SCSolarZenithAngle
VIIRS-MOD-GEO-TC_All/SCVelocity
VIIRS-MOD-GEO-TC_All/SatelliteAzimuthAngle
VIIRS-MOD-GEO-TC_All/SatelliteRange
VIIRS-MOD-GEO-TC_All/SatelliteZenithAngle
VIIRS-MOD-GEO-TC_All/SolarAzimuthAngle
VIIRS-MOD-GEO-TC_All/SolarZenithAngle
VIIRS-MOD-GEO-TC_All/StartTime
Data_Products/VIIRS-MOD-GEO-TC/VIIRS-MOD-GEO-TC_Aggr
Data_Products/VIIRS-MOD-GEO-TC/VIIRS-MOD-GEO-TC_Gran_0
[u'VIIRS-MOD-GEO-TC_All/Height', u'VIIRS-MOD-GEO-TC_All/Latitude', u'VIIRS-MOD-GEO-TC_All/Longitude', u'VIIRS-MOD-GEO-TC_All/MidTime', u'VIIRS-MOD-GEO-TC_All/ModeGran', u'VIIRS-MOD-GEO-TC_All/ModeScan', u'VIIRS-MOD-GEO-TC_All/NumberOfScans', u'VIIRS-MOD-GEO-TC_All/PadByte1', u'VIIRS-MOD-GEO-TC_All/QF1_SCAN_VIIRSSDRGEO', u'VIIRS-MOD-GEO-TC_All/QF2_SCAN_VIIRSSDRGEO', u'VIIRS-MOD-GEO-TC_All/QF2_VIIRSSDRGEO', u'VIIRS-MOD-GEO-TC_All/SCAttitude', u'VIIRS-MOD-GEO-TC_All/SCPosition', u'VIIRS-MOD-GEO-TC_All/SCSolarAzimuthAngle', u'VIIRS-MOD-GEO-TC_All/SCSolarZenithAngle', u'VIIRS-MOD-GEO-TC_All/SCVelocity', u'VIIRS-MOD-GEO-TC_All/SatelliteAzimuthAngle', u'VIIRS-MOD-GEO-TC_All/SatelliteRange', u'VIIRS-MOD-GEO-TC_All/SatelliteZenithAngle', u'VIIRS-MOD-GEO-TC_All/SolarAzimuthAngle', u'VIIRS-MOD-GEO-TC_All/SolarZenithAngle', u'VIIRS-MOD-GEO-TC_All/StartTime', u'Data_Products/VIIRS-MOD-GEO-TC/VIIRS-MOD-GEO-TC_Aggr', u'Data_Products/VIIRS-MOD-GEO-TC/VIIRS-MOD-GEO-TC_Gran_0']


I'm not sure if the geolocation variables can be accessed through scripting unless they are in a combined file. The programmers will have to take a look at that as well.

If you have any other questions or comments, just let us know.

Thanks, Jay
User avatar
hproe
Posts: 504
Joined: Sat Nov 27, 2010 3:46 pm

Re: VIIRS metadata retrieval

Post by hproe »

Hi Jay -

Many thanks for launching an inquiry for expanding the VIIRS methods. Giving this some priority would be welcome.

I agree with your point 1.

As regards your point 2, I continue to see differences (latest nightly) between listing with listVIIRFieldsInFile() and the Fields list in the chooser. In particular 'MoonIllumFraction' is missing in Fields, when comparing the two lists, probably an oversight as the other paramteres of sun and moon are available:
Fields.png

while in the jython Shell 'MoonIllumFraction' is there:
listVIIRSFieldsInFile('/Users/hproe_000/Documents/DATA/VIIRSdnbRGB/GDNBO-SVDNB_npp_d20161211_t0108092_e0113478_b26540_c20161230130535507304_noaa_ops.h5')
VIIRS-DNB-GEO_All/Height
VIIRS-DNB-GEO_All/Height_TC
VIIRS-DNB-GEO_All/Latitude
VIIRS-DNB-GEO_All/Latitude_TC
VIIRS-DNB-GEO_All/Longitude
VIIRS-DNB-GEO_All/Longitude_TC
VIIRS-DNB-GEO_All/LunarAzimuthAngle
VIIRS-DNB-GEO_All/LunarZenithAngle
VIIRS-DNB-GEO_All/MidTime
VIIRS-DNB-GEO_All/ModeGran
VIIRS-DNB-GEO_All/ModeScan
VIIRS-DNB-GEO_All/MoonIllumFraction
VIIRS-DNB-GEO_All/MoonPhaseAngle
VIIRS-DNB-GEO_All/NumberOfScans
VIIRS-DNB-GEO_All/PadByte1
VIIRS-DNB-GEO_All/QF1_SCAN_VIIRSSDRGEO
VIIRS-DNB-GEO_All/QF2_SCAN_VIIRSSDRGEO
VIIRS-DNB-GEO_All/QF2_VIIRSSDRGEO
VIIRS-DNB-GEO_All/QF2_VIIRSSDRGEO_TC
VIIRS-DNB-GEO_All/SCAttitude
VIIRS-DNB-GEO_All/SCPosition
VIIRS-DNB-GEO_All/SCSolarAzimuthAngle
VIIRS-DNB-GEO_All/SCSolarZenithAngle
VIIRS-DNB-GEO_All/SCVelocity
VIIRS-DNB-GEO_All/SatelliteAzimuthAngle
VIIRS-DNB-GEO_All/SatelliteRange
VIIRS-DNB-GEO_All/SatelliteZenithAngle
VIIRS-DNB-GEO_All/SolarAzimuthAngle
VIIRS-DNB-GEO_All/SolarZenithAngle
VIIRS-DNB-GEO_All/StartTime
VIIRS-DNB-SDR_All/ModeGran
VIIRS-DNB-SDR_All/ModeScan
VIIRS-DNB-SDR_All/NumberOfBadChecksums
VIIRS-DNB-SDR_All/NumberOfDiscardedPkts
VIIRS-DNB-SDR_All/NumberOfMissingPkts
VIIRS-DNB-SDR_All/NumberOfScans
VIIRS-DNB-SDR_All/PadByte1
VIIRS-DNB-SDR_All/QF1_VIIRSDNBSDR
VIIRS-DNB-SDR_All/QF2_SCAN_SDR
VIIRS-DNB-SDR_All/QF3_SCAN_RDR
VIIRS-DNB-SDR_All/Radiance
Data_Products/VIIRS-DNB-GEO/VIIRS-DNB-GEO_Aggr
Data_Products/VIIRS-DNB-GEO/VIIRS-DNB-GEO_Gran_0
Data_Products/VIIRS-DNB-GEO/VIIRS-DNB-GEO_Gran_1
Data_Products/VIIRS-DNB-GEO/VIIRS-DNB-GEO_Gran_2
Data_Products/VIIRS-DNB-GEO/VIIRS-DNB-GEO_Gran_3
Data_Products/VIIRS-DNB-SDR/VIIRS-DNB-SDR_Aggr
Data_Products/VIIRS-DNB-SDR/VIIRS-DNB-SDR_Gran_0
Data_Products/VIIRS-DNB-SDR/VIIRS-DNB-SDR_Gran_1
Data_Products/VIIRS-DNB-SDR/VIIRS-DNB-SDR_Gran_2
Data_Products/VIIRS-DNB-SDR/VIIRS-DNB-SDR_Gran_3
[u'VIIRS-DNB-GEO_All/Height', u'VIIRS-DNB-GEO_All/Height_TC', u'VIIRS-DNB-GEO_All/Latitude', u'VIIRS-DNB-GEO_All/Latitude_TC', u'VIIRS-DNB-GEO_All/Longitude', u'VIIRS-DNB-GEO_All/Longitude_TC', u'VIIRS-DNB-GEO_All/LunarAzimuthAngle', u'VIIRS-DNB-GEO_All/LunarZenithAngle', u'VIIRS-DNB-GEO_All/MidTime', u'VIIRS-DNB-GEO_All/ModeGran', u'VIIRS-DNB-GEO_All/ModeScan', u'VIIRS-DNB-GEO_All/MoonIllumFraction', u'VIIRS-DNB-GEO_All/MoonPhaseAngle', u'VIIRS-DNB-GEO_All/NumberOfScans', u'VIIRS-DNB-GEO_All/PadByte1', u'VIIRS-DNB-GEO_All/QF1_SCAN_VIIRSSDRGEO', u'VIIRS-DNB-GEO_All/QF2_SCAN_VIIRSSDRGEO', u'VIIRS-DNB-GEO_All/QF2_VIIRSSDRGEO', u'VIIRS-DNB-GEO_All/QF2_VIIRSSDRGEO_TC', u'VIIRS-DNB-GEO_All/SCAttitude', u'VIIRS-DNB-GEO_All/SCPosition', u'VIIRS-DNB-GEO_All/SCSolarAzimuthAngle', u'VIIRS-DNB-GEO_All/SCSolarZenithAngle', u'VIIRS-DNB-GEO_All/SCVelocity', u'VIIRS-DNB-GEO_All/SatelliteAzimuthAngle', u'VIIRS-DNB-GEO_All/SatelliteRange', u'VIIRS-DNB-GEO_All/SatelliteZenithAngle', u'VIIRS-DNB-GEO_All/SolarAzimuthAngle', u'VIIRS-DNB-GEO_All/SolarZenithAngle', u'VIIRS-DNB-GEO_All/StartTime', u'VIIRS-DNB-SDR_All/ModeGran', u'VIIRS-DNB-SDR_All/ModeScan', u'VIIRS-DNB-SDR_All/NumberOfBadChecksums', u'VIIRS-DNB-SDR_All/NumberOfDiscardedPkts', u'VIIRS-DNB-SDR_All/NumberOfMissingPkts', u'VIIRS-DNB-SDR_All/NumberOfScans', u'VIIRS-DNB-SDR_All/PadByte1', u'VIIRS-DNB-SDR_All/QF1_VIIRSDNBSDR', u'VIIRS-DNB-SDR_All/QF2_SCAN_SDR', u'VIIRS-DNB-SDR_All/QF3_SCAN_RDR', u'VIIRS-DNB-SDR_All/Radiance', u'Data_Products/VIIRS-DNB-GEO/VIIRS-DNB-GEO_Aggr', u'Data_Products/VIIRS-DNB-GEO/VIIRS-DNB-GEO_Gran_0', u'Data_Products/VIIRS-DNB-GEO/VIIRS-DNB-GEO_Gran_1', u'Data_Products/VIIRS-DNB-GEO/VIIRS-DNB-GEO_Gran_2', u'Data_Products/VIIRS-DNB-GEO/VIIRS-DNB-GEO_Gran_3', u'Data_Products/VIIRS-DNB-SDR/VIIRS-DNB-SDR_Aggr', u'Data_Products/VIIRS-DNB-SDR/VIIRS-DNB-SDR_Gran_0', u'Data_Products/VIIRS-DNB-SDR/VIIRS-DNB-SDR_Gran_1', u'Data_Products/VIIRS-DNB-SDR/VIIRS-DNB-SDR_Gran_2', u'Data_Products/VIIRS-DNB-SDR/VIIRS-DNB-SDR_Gran_3']


I have uploaded the GDBNO-SVDNB file as given in the list command above on the SSEC server.

cheers, HP
User avatar
jayh
Posts: 424
Joined: Thu Jan 15, 2009 10:34 pm

Re: VIIRS metadata retrieval

Post by jayh »

Hi HP-

I see the differences now, I overlooked them. I made a separate inquiry to get these types of variables available in the Field Selector, Inq. 2477.

Thanks, Jay
User avatar
hproe
Posts: 504
Joined: Sat Nov 27, 2010 3:46 pm

Re: VIIRS metadata retrieval

Post by hproe »

Hi -

Many thanks to Jay for Inq. 2477. I have now implemented a formula that adjusts the DNB signal to near-constant contrast (NCC) independent of moon/sun zenith angle (the adjustment algorithm is not my invention, it comes from a paper to be published!). It is sad that, in order to run the formula, the 'moon illumination fraction' has to be read externally from the file with a netCDF tool and input manually when calling the formula.
Also, I still miss the Jython methods to interrogate all fields in the file and not only time, see Inq. 2475. This hinders me to fully script the NCC method.

HP
Post Reply