HDF and NetCDF support in McIDAS-V

[joleenf]

I am using an AIRS file in McIDAS-V. While it can be read into McIDAS-V, some of the fields are missing.

1.) If this file can be fixed in NcML, can this same ability be transferred to McV. TomR felt the major disadvantage of this is the process would have to be repeated for every new file. It would be much easier to save a wrapper like AIRS1.01format which is used for every file of that format. One user had mentioned they would like a GUI type interface which allows them to point to what is lat/lon and perhaps define for McIDAS-V other fields which it has not been able to recognize or at least fix what is wrong. I think that TomR had mentioned this is a very difficult task because of the way HDF format is being used and has been constructed, but he will have to confirm that for me. I might have misunderstood.

2.) This is related to issue 1, a user had mentioned how frustrating and expensive it can be to pay for a new server for every new hdf file. This user decided to use IDL as an answer to this problem. I realize that UNIDATA has set some really nice standards at it would be very nice if everyone complied or that we could request that people comply. However, this is not the reality. Users will be trying to put their hdf files into McIDAS-V and they will get frustrated because they probably won't be compliant files. This is contentious: can McIDAS-V be flexible or informative enough to allow a fix upon loading the file which then can be saved as a wrapper for all files of that type? I think there is a similar paradigm with point data, though it has been a while since I worked with that. I am certain users will be frustrated if they have to fix their file in another program before using it in McIDAS-V. I would think they would be much more likely to continuing processing their data in another package such as IDL and MatLab if they find it easier to get their data into it initially.

3.) Kaba, TomW. and I have all spoke of the Time variable in the header, that is another post. We had also talked about assigning lat/lons to file which for whatever reason have not had a lat/lon dimension added to them. For instance, if there is a file that over Alaska and the end points or center points are known, is there a way to assign those values to the file when it is read into McIDAS-V so that it can be displayed over Alaska?

I think all these things can be achieved in NcML now, is there a way to fold it into McIDAS-V? Would user's find it useful as is, or would there need to be some change to the NcML interface to make it clearer to a novice user?

Joleen

[kaba]

Joleen,
I remember a little over a year ago I worked with Kathy on converting the AIRS DPI .hdf files to McIDAS-V netCDF and AWIPS netCDF files.
I am wondering if it is these files that your trying to read into McIDAS-V and some variables were simply not re-written in the netCDF versions.
If that is the case, it should be an easy fix.

[joleenf]

Kaba,

These are the same files with additional fields. However, they are not being generated as netCDF, instead they are hdf4. I am not sure if Kathy "repair" something within the hdf file so that McIDAS-V can read all the fields or if the hdf reader needs something extra to allow it to read the extra fields. My general question is how to avoid this cycle every time a new field is added to a file. This will be an easy fix because Kathy is willing to work at the source and I can ask you for additional help. Change the situation a little and pretend that I am not at CIMSS, the person who is providing the data is unwilling to change it and I still want to bring it into McIDAS-V. What is the best way to approach this problem and can it be easier to do within McIDAS-V or will it always be something which needs to be fixed outside of McIDAS-V before attempting to load into the file selector. Since hdf files are self-described, a user may be able to tell McIDAS-V how to use the fields, but they need that option.

Joleen

[nickb]

Hi Joleen, I'm currently working on a "reader" for NetCDF files which may solve some of these problems. While this is for NetCDF, the NetCDF-Java library comes with support for some other file types (including HDF), so much of this should cross over.

A quick overview:

1. An NcML editor for the user's selected file, which allows saving to NetCDF.
2. A basic 2d-field viewer (displays data on a x/y grid.)
3. Allow the user to specify lat/lon bounds (or variables) and a data variable, and attempt to create a McV display from those.

The first two are working, but I got hung up on the third and then distracted by my other projects. I'll set aside some time this week and next and see if I can't get a "first-cut" version going for testing.

[joleenf]

Hi Nick,

That sounds great!

Joleen

[kbedka1]

I definitely appreciate Joleen's statement:

"Change the situation a little and pretend that I am not at CIMSS, the person who is providing the data is unwilling to change it and I still want to bring it into McIDAS-V. What is the best way to approach this problem and can it be easier to do within McIDAS-V or will it always be something which needs to be fixed outside of McIDAS-V before attempting to load into the file selector. Since hdf files are self-described, a user may be able to tell McIDAS-V how to use the fields, but they need that option."

Here at NASA LaRC, I've run in numerous situations where a user wants to look at their HDF format data (both HDF4 and HDF5) in McIDAS-V but cannot because their instrument or model data is not yet supported. I look forward to any new development in Mc-V that will allow more flexibility with input data format/structure. Though getting data providers to conform to standards is a worthy goal at the current and in the future, the community has petabytes of valuable, but poorly structured archived data they want to analyze.

[joleenf]

Nick,

I heard that the reader is partially in the nightly builds but have not been able to use it. What type of file will force the reader to appear? I have used a few different hdf files which I know need some manipulation, but have been forced to change the contents outside of McV.

Joleen

[tomw]

Joleen -- Aim the 'File Chooser' at your HDF file, and set the "type" to "netcdf grids" (or whatever it really is)...if the file's contents cannot be understood, Nick's tool should pop up....at least it did for me when I was working with Kathy's file. BTW, I already filed a feature request to support 3D arrays (2 for nav, and one to select as an "index"). There are a few other issues that will need to evolve, but the more feedback Nick can get, the better!

[joleenf]

Hi Tom,

I tried this today and noticed different behavior. I recently updated my nightly build (9/6/2011). All the hdf files I had been using will load into McIDAS-V. This includes the UW AIRS SFOV direct broadcast retrieval, the ash product (GEOCAT) and the convective initiation/ctc products. The cloud top cooling product is displaying correctly, but there are still problems with the AIRS file. Attached is an example. First, I can only zoom to the level seen in this gif. After this zoom level, the display becomes blank. Second, the missing data is still not handled well. It would be nice to have control at the point of loading the data into the field selector, where a user could specify and fix fields which are likely to be a problem (degrees_north and degrees_east are not being used in this hdf file) If at the time data is displayed, it would also be nice to be able to specify missing values. I think the problem for this file is that the missing value is not specified at all.

Joleen

[kaba]

Just to add on to what Joleen said. We were just looking at the AIRS data in McIDAS-V nightly built together and the data
loaded without errors but the display had some issues, probably related to the missing values in this case.
It would be nice to be able to pop up a GUI when such things happens where you can set the missing values, range etc and
then reload the data if possible.