Some Questions

[carlospsneto]

Hello,

I'm trying to learn McIDAS-V Scripting. I'm having some troubles.

Question 1. I created a local dataset called "November28". In this directory i have images of all day, but and when i try to acess this dataset using:

Code: Select all

      desc=getLocalADDEEntry(dataset='November28',imageType = 'MSG')

And list tha data using:

Code: Select all

 directoryList = listADDEImages(localEntry=desc, band=9, unit='TEMP')
print directoryList

I get this message:

Code: Select all

"[{'center-latitude-resolution': 3.01725, 'source-type': 'MSG', 'calibration-type': 'RAW', 'descriptor': u'FD', 'accounting': ('idv', '0'), 'lines': 3712, 'band-count': 1, 'sensor-id': 52, 'calibration-unit-name': None, 'bandNumber': 9, 'centerLocation': (0.0, -0.0), 'directory-block': [96, 4, 52, 110332, 234500, 3, 3, 1, 3712, 3712, 2, 3, 3, 1, 0, 0, 110332, 234500, 256, 0, 0, 0, 0, 0, 1297303328, 1684108385, 543584879, 1830836306, 1230250070, 840966176, 538976288, 538976288, 0, 2368, 256, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 110332, 234500, 0, 0, 0, 0, 1297303328, 1380013856, 2, 0, 0, 0, 0, 0, 0, 0, 0, 768, 50], 'calibration-scale-factor': 1, 'day': '2010332', 'url': u'adde://localhost/imagedirectory?&PORT=8112&COMPRESS=gzip&USER=idv&PROJ=0&VERSION=1&DEBUG=false&TRACE=0&GROUP=CCM&DESCRIPTOR=FD&BAND=9&UNIT=TEMP&POS=0', 'center-longitude': -0.0, 'center-longitude-resolution': 2.99704, 'center-latitude': 0.0, 'debug': 'false', 'resolution': (3.01725, 2.99704), 'server': u'localhost', 'imageSize': (3712, 3712), 'unitType': 'TEMP', 'sensor-type': 'MSG-2', 'elements': 3712, 'dataset': u'CCM', 'bandList': [9], [b]'start-time': 2010-11-28 23:45:00Z[/b], 'memo-field': 'MSG data from XRIT V2           ', 'calinfo': ['RAW', 'RAW', 'RAD', 'RADIANCE', 'TEMP', 'TEMPERATURE', 'BRIT', 'BRIGHTNESS'], 'time': '23:45:00', 'nominal-time': 2010-11-28 23:45:00Z, 'bands': [9], 'unitList': ['TEMP']}]"

Note that the start-time 23:45:00Z is actually the final time that i have in the data. What am I doing wrong? Why i cannot access the other time periods?



Question 2. How do i open .nc files using scripting?

Question 3. Is there a simple way to use the predefined formulas (Field Selector->Data Sourcer->Formulas) on scripting?
For a exemplo the Mask Function formula.


Thank you.

[bobc]

Hello carlospsneto -

1. By default, listADDEImages only gets the 1 most recent image, which is why you are only seeing the last image in your dataset returned. From the documentation of listADDEImages, the default value for the position= keyword is 0. This only gives you the most recent image. If you want to get a different image or different range of images, you would need to tack on position= to your listADDEImages command.

ex. directoryList = listADDEImages(localEntry=desc, band=9,position=(-9,0), unit='TEMP')
- This will get you the 10 most recent images.

2. It is currently not possible to open .nc files through scripting, but this is on our priority list for a future release.

3. Here is an example of how you could use the mask formula to only display values greater than 160:

Code: Select all

# Creating local ADDE dataset
dataPath = '/Users/rcarp/area/'
makeLocalADDEEntry(dataset='AREA2', save=True, format="McIDAS Area",
    mask=dataPath, imageType='AREA files')

# Setting the descriptor
desc=getLocalADDEEntry(dataset='AREA2',imageType='AREA files')

# Creating dictionary of parameters to pass into listADDEImages
adde_parms = dict(
    debug=True,
    server='localhost',
    dataset='AREA2',
    size='ALL',
    localEntry=desc,
    mag=(1, 1),
    unit='BRIT',
    band=1,
)

# listADDEImages command
directoryList=listADDEImages(descriptor=desc,**adde_parms)
print directoryList

# passing output of listADDEImages into getADDEImage
for imageDir in directoryList:
    metaData,data=getADDEImage(time=imageDir['time'],**adde_parms)

# Mask
maskData=mask(data,'>',160,1)
panel=buildWindow(height=600,width=800)
layer=panel[0].createLayer('Image Display',maskData)

This will create a display with values of only 0 and 1. 1 will display where the mask is true (where the brightness values are greater than 160), and 0 will display where the mask value is false (where the brightness value is less than 160). If you wish to apply the mask to a display of a satellite image, you can do the following:

Code: Select all

product=mul(maskData,data)
panel=buildWindow(height=600,width=800)
layer=panel[0].createLayer('Image Display',product)
layer.setEnhancement('Inverse Gray Scale',range=(160,310))
layer.setColorScale(visible=True, placement='Left',font='SansSerif',size=18,showUnit=True)
panel[0].setWireframe(False)

This will multiply the result of your mask with the data returned from getADDEImage. I created a variable called 'product' which is the result of 'maskData' multiplied by 'data'. 'product' is then used in the createLayer command.

Hopefully this gets you started. If you have any questions, please let us know.

Thanks -
Bob Carp
McIDAS User Services

[ghansham]

Regarding question no. 2, you can use
makeDatasource (ncfile).
Pretty straightforward.
If the file is gridded data you should
See a datasource object getting returned
successfully. I hope I am not misunderstanding
your query. See documentation for more details.
Ghansham

[carlospsneto]

Thank you,

I could successfully made a mask.

But, i still have some doubts about the time sequence. How can i display a image display in a time sequence?
I tryed do this:

Code: Select all

metadata, data=getADDEImage(localEntry=desc,size='ALL', mag=(1,1),band=9,unit='TEMP',time=('21:00:00','22:15:00'))
panel=buildWindow()
dataLayer = panel[0].createLayer('Image Display', data)

hoping that it could result in a time sequence display of images:(21:00:00-21:15:00-21:30:00-21:45:00-22:00:00-22:15:00).
But only was displayed the final image 22:15:00. After that, I tryed:

Code: Select all

dataLayer = panel[0].createLayer('Image Sequence Display', data)

and the result was the same, was only displayed only the final time.

I will try makeDatasource (ncfile)

[joleenf]

Regarding #2, I have a few tools in my jython library:

dirReadExtractTime.jar

(1.36 KiB) Downloaded 322 times

see http://www.ssec.wisc.edu/mcidas/doc/mcv ... ugins.html in the user's guide for installing a plugin. You will need to reboot after installation. After installation of the plugin, some of my user code will appear in your jython library.

This is roughly how they are used, you would be most interested in using #3.
1.) fileSearch - scans specified directory looking for a file pattern. Returns a list of fileNames.
example:
listFileNames = fileSearch("<pathname>", "<regex>")

so
listFileNames = fileSearch("<pathname>", "geocat*.hdf")

will return a list of geocat fies in pathname

use that list to extract a time from the file name as a java.util.Date

2.) extractTimeFromName - returns a java.util.Date String from the fileName (formatting procedure is slightly different than for visad.DateTime, so you might want to convert again)
example:
fileName = '<pathname>' + 'geocatL2.GOES-14.2013138.2045.hdf'
fileDate = extractTimeFromName(fileName, demarkPattern="#geocatL2.GOES-14.#yyyymmdd.HHmmss")

it is important that the "#" signs bracket the portion of the fileName before the date portion. Everything after the date portion of the fileName will be ignored.

3.) getGridFlatField - extracts a flatField from an hdf file
example:
dataGrid, dataDescription = getGridFlatField(fileName, longname, sourceType='netcdf.grid')
dataGrid, dataDescription = getGridFlatField(fileName, 'Temperature', sourceType='netcdf.grid')

The last routine is the one which reads the hdf/netCDF file. It is important to know the longname for the grid to be used. The sourceType will be netcdf.grid for both hdf and netcdf file types. This will not work on Suomi NPP data.


I hope this helps.

Joleen

[carlospsneto]

Hello Joleen,

Thank for your help. I installed the plugin and tryed:

Code: Select all

dataGrid, dataDescription = getGridFlatField('C:\Users\Carlos\Downloads\Nova pasta (2)\ tropopause_temp.nc', 'mean Daily Air Temperature at Tropopause', sourceType='netcdf.grid')

And the result was this error message
None
ERROR:
'NoneType' object has no attribute 'findDataChoice'
Traceback (most recent call last):
File "", line 1, in
TypeError: 'int' object is not iterable

Am I missing something?

[mhiley]

Hi Carlos,

It looks to me like "makeDataSource" is failing in Joleen's getGridFlatField (it returns a null, which is why the script fails at "findDataChoice").

I suspect the path to the tropopause_temp.nc file is either incorrect, or something along the way doesn't like the spaces or parentheses in the pathname... anyway, you can confirm whether this is the issue using "os.path.exists":

Code: Select all

from os.path import exists
print exists('/path/to/your/file')

-Mike

[bobc]

hello -

As far as creating a loop of images, here is a sample script you can use to get started:

Code: Select all

# Creating local ADDE dataset
dataPath = '/Users/rcarp/area/'
makeLocalADDEEntry(dataset='AREA2', save=True, format="McIDAS Area",
    mask=dataPath, imageType='AREA files')

# Setting the descriptor
desc=getLocalADDEEntry(dataset='AREA2',imageType='AREA files')

# Creating dictionary of parameters to pass into listADDEImages
adde_parms = dict(
    debug=True,
    server='localhost',
    dataset='AREA2',
    size='ALL',
    localEntry=desc,
    mag=(1, 1),
    unit='BRIT',
    band=1,
    position='ALL',
)

# listADDEImageTimes command
directoryList=listADDEImageTimes(time=('20:31:00','20:36:00'),descriptor=desc,**adde_parms)
print directoryList

imageLoop=[]

# passing output of listADDEImageTimes into getADDEImage
for imageDir in directoryList:
    metaData,data=getADDEImage(time=imageDir['time'],**adde_parms)
    imageLoop.append(data)

# Build window and create loop
panel=buildWindow(height=600,width=800)
layer=panel[0].createLayer('Image Sequence Display', imageLoop)
layer.setEnhancement('Gray Scale',range=(0,255))
layer.setColorScale(visible=True, placement='Left',font='SansSerif',size=18,showUnit=True)
panel[0].setWireframe(False)

- Bob

[joleenf]

Hi Mike,

It appears that the file is not being found. You could try each line from that subroutine in the jython shell

Code: Select all

dataSource1 = makeDataSource('C:\Users\Carlos\Downloads\Nova pasta (2)\ tropopause_temp.nc', 'netcdf.grid')

print dataSource1

dataChoice1 = dataSource1.findDataChoice('mean Daily Air Temperature at Tropopause')

print dataChoice1

To find out which portion is failing. It seems to me from the error that you see, the netCDF file was not properly found. Is there really a space between "\" and "tropopause_temp.nc"?

Joleen

[Jon]

Hi Carlos,

It looks to me like "makeDataSource" is failing in Joleen's getGridFlatField (it returns a null, which is why the script fails at "findDataChoice").

I suspect the path to the tropopause_temp.nc file is either incorrect, or something along the way doesn't like the spaces or parentheses in the pathname... anyway, you can confirm whether this is the issue using "os.path.exists":

Code: Select all

from os.path import exists
print exists('/path/to/your/file')

-Mike

Incredibly, it seems that NetCDF doesn't like spaces in its filenames! Check out the output from ncdump (and yes, both files exist); it just drops everything after the space character:

Code: Select all

print ncdump('/Volumes/LaCIE/mcidasv/hydra/IASI/IASI_xxx_1C_M02_20070115_1140.nc')
netcdf /Volumes/LaCIE/mcidasv/hydra/IASI/IASI_xxx_1C_M02_20070115_1140 {
  dimensions:
    obsChannelIndex = 8461;
    obsElement = 52;
    obsLine = 80;
  variables:
    float observations(obsLine, obsElement, obsChannelIndex);
    float obsLatitude(obsLine, obsElement);
    float obsLongitude(obsLine, obsElement);
    float observationChannels(obsChannelIndex);
  // global attributes:
  :instrumentName = "IASI";
 data:
}
None
print ncdump('/Volumes/LaCIE/mcidasv/hydra/IASI/IASI_xxx_1C_M02_20070115 1140.nc')
file not found= /Volumes/LaCIE/mcidasv/hydra/IASI/IASI_xxx_1C_M02_20070115
None