differences in behavior, aggregated vs single file

Errors and unexpected results
User avatar
ghansham
Posts: 170
Joined: Thu Nov 11, 2010 5:40 pm

Re: differences in behavior, aggregated vs single file

Post by ghansham »

How you are aggregating files?
User avatar
bobc
Posts: 938
Joined: Mon Nov 15, 2010 5:57 pm

Re: differences in behavior, aggregated vs single file

Post by bobc »

Hi Kris -

We heard back from Unidata and here was what they said:
The problem with this is that the Java doesn't support unsigned type, so the valid range is -128 to 127. When loading a single file, the type being converted to short in the process ( I need to look into this later), this is why it is showing all the values. I think the NcML wrapper is a good solution, we can even create a plugin with this wrapper and the user only see a new data source type. Otherwise, we need to wait till we all move to netcdf java 5.0 which does support unsigned type.

The ncml time aggregate should be ok in the 5.0 as long as there is only one time dimension in the dataset.

Using the files you posted on ftp, we investigated this plugin approach and came up with this plugin:
unsigned2short.jar
Plugin
(1.04 KiB) Downloaded 239 times

With this plugin installed, from the local file chooser you will see a new data type of 'Aggregate Grids by Time (unsigned to short)'. Select this data type, your files, and click Add Source. Display the 'Anvil Cloud Detection ID' field and you should see that all of the data values display (just like things work with the NcML wrapper posted earlier).

This plugin is composed of 2 files:
datasource.xml
(466 Bytes) Downloaded 286 times

short.ncml
(263 Bytes) Downloaded 295 times

The datasource.xml file is how the new 'Aggregate Grids by Time (unsigned to short)' data type was created. There's one line in that file referencing a 'ncmltemplate' file as short.xml. short.xml is where we are setting the "ir_anvil_detection" variable as a short (just as was done with the wrapper file).

You mentioned that you have lots of arrays of this type. I believe the short.xml file can be made much more robust to handle additional variables, but I don't have any data to test with. Does working with this new data type (via the plugin) sound like an acceptable solution to this problem before we update the NetCDF/java version? If it does, can you please send us some files that have multiple unsigned variables that give you the problem (and let us know the variable names)? I believe that this plugin can be made much more robust to handle more variables than just 'ir_anvil_detection'.

Thanks -
Bob Carp
User avatar
ghansham
Posts: 170
Joined: Thu Nov 11, 2010 5:40 pm

Re: differences in behavior, aggregated vs single file

Post by ghansham »

Hi All

Can somebody try this formula "Miscellaneous->Make a time sequence from single time grids/images"
Or even you can try "Grids->Time Steps->Merge Time Sequences"
but without the 'Aggregate Grids by Time (unsigned to short)' jar.

It is giving the expected behavior. I mean the values greater than 127 are appearing.

I tried it with an older version of IDV (4.1). It works well.


regards
Ghansham
User avatar
jayh
Posts: 415
Joined: Thu Jan 15, 2009 10:34 pm

Re: differences in behavior, aggregated vs single file

Post by jayh »

Hi Kris/Ghansham-

We have used those formulas, and they do work, however when you have more than 4 files and want longer time sequences, the process is very tedious to choose those variables. That is why we would like to get the aggregation working.

We do have some good news, Yuan at Unidata made a code change in the CDM (common data model) and has checked in a temporary fix that will we be testing soon. He doesn't want this to be a long term solution for IDV/McIDAS-V, but he thinks it will work before he can discuss a better fix with his CDM colleagues.

I'll keep updating this thread as we work on getting this change merged into a McV nightly.

Thanks, Jay
User avatar
ghansham
Posts: 170
Joined: Thu Nov 11, 2010 5:40 pm

Re: differences in behavior, aggregated vs single file

Post by ghansham »

Hi

It would be pretty dumb at my part still I am unable to reproduce the bug, I mean can you explain how you are aggregating the files when it shows wrong output.

Regards
Ghamsham
User avatar
jayh
Posts: 415
Joined: Thu Jan 15, 2009 10:34 pm

Re: differences in behavior, aggregated vs single file

Post by jayh »

Hi Ghansham-

Load the grids through the General Files chooser. At the top choose Data Type of "Aggregate Grids by Time", select all four files, and Add Source. This will make one Data Source in the field selector. Display the Anvil Cloud Detection with color shaded plan view and that should show the error.

Thanks, Jay
User avatar
bobc
Posts: 938
Joined: Mon Nov 15, 2010 5:57 pm

Re: differences in behavior, aggregated vs single file

Post by bobc »

Hi Kris -

We merged Unidata's changes into McIDAS-V, and if you pick up today's (12/6) 1.7 beta1 nightly you should see that the aggregated behavior of the 'Anvil Cloud Detection ID' is now behaving better. All of the data values probe and plot, not just those less than 127, so there is no need to use the plugin I posted earlier. If you give this a look and notice any problems, please let us know.

Thanks -
Bob
Post Reply