McIDAS-V Support Forums

Support and discussion of the McIDAS-V software package
Forum HomeMcIDAS-V Home
It is currently 04 Aug 2020, 02:56

All times are UTC




Post new topic Reply to topic  [ 17 posts ]  Go to page Previous  1, 2
Author Message
 Post subject: Re: differences in behavior, aggregated vs single file
PostPosted: 30 Nov 2016, 04:06 
User avatar

Joined: 11 Nov 2010, 17:40
Posts: 168
How you are aggregating files?


Top
 Profile  
 
 Post subject: Re: differences in behavior, aggregated vs single file
PostPosted: 30 Nov 2016, 17:23 
User avatar

Joined: 15 Nov 2010, 17:57
Posts: 917
Hi Kris -

We heard back from Unidata and here was what they said:
Quote:
The problem with this is that the Java doesn't support unsigned type, so the valid range is -128 to 127. When loading a single file, the type being converted to short in the process ( I need to look into this later), this is why it is showing all the values. I think the NcML wrapper is a good solution, we can even create a plugin with this wrapper and the user only see a new data source type. Otherwise, we need to wait till we all move to netcdf java 5.0 which does support unsigned type.

The ncml time aggregate should be ok in the 5.0 as long as there is only one time dimension in the dataset.

Using the files you posted on ftp, we investigated this plugin approach and came up with this plugin:
Attachment:
File comment: Plugin
unsigned2short.jar [1.04 KiB]
Downloaded 189 times

With this plugin installed, from the local file chooser you will see a new data type of 'Aggregate Grids by Time (unsigned to short)'. Select this data type, your files, and click Add Source. Display the 'Anvil Cloud Detection ID' field and you should see that all of the data values display (just like things work with the NcML wrapper posted earlier).

This plugin is composed of 2 files:
Attachment:
datasource.xml [466 Bytes]
Downloaded 238 times

Attachment:
short.ncml [263 Bytes]
Downloaded 243 times

The datasource.xml file is how the new 'Aggregate Grids by Time (unsigned to short)' data type was created. There's one line in that file referencing a 'ncmltemplate' file as short.xml. short.xml is where we are setting the "ir_anvil_detection" variable as a short (just as was done with the wrapper file).

You mentioned that you have lots of arrays of this type. I believe the short.xml file can be made much more robust to handle additional variables, but I don't have any data to test with. Does working with this new data type (via the plugin) sound like an acceptable solution to this problem before we update the NetCDF/java version? If it does, can you please send us some files that have multiple unsigned variables that give you the problem (and let us know the variable names)? I believe that this plugin can be made much more robust to handle more variables than just 'ir_anvil_detection'.

Thanks -
Bob Carp


Top
 Profile  
 
 Post subject: Re: differences in behavior, aggregated vs single file
PostPosted: 01 Dec 2016, 05:54 
User avatar

Joined: 11 Nov 2010, 17:40
Posts: 168
Hi All

Can somebody try this formula "Miscellaneous->Make a time sequence from single time grids/images"
Or even you can try "Grids->Time Steps->Merge Time Sequences"
but without the 'Aggregate Grids by Time (unsigned to short)' jar.

It is giving the expected behavior. I mean the values greater than 127 are appearing.

I tried it with an older version of IDV (4.1). It works well.


regards
Ghansham


Top
 Profile  
 
 Post subject: Re: differences in behavior, aggregated vs single file
PostPosted: 01 Dec 2016, 18:20 
User avatar

Joined: 15 Jan 2009, 22:34
Posts: 408
Hi Kris/Ghansham-

We have used those formulas, and they do work, however when you have more than 4 files and want longer time sequences, the process is very tedious to choose those variables. That is why we would like to get the aggregation working.

We do have some good news, Yuan at Unidata made a code change in the CDM (common data model) and has checked in a temporary fix that will we be testing soon. He doesn't want this to be a long term solution for IDV/McIDAS-V, but he thinks it will work before he can discuss a better fix with his CDM colleagues.

I'll keep updating this thread as we work on getting this change merged into a McV nightly.

Thanks, Jay


Top
 Profile  
 
 Post subject: Re: differences in behavior, aggregated vs single file
PostPosted: 02 Dec 2016, 01:21 
User avatar

Joined: 11 Nov 2010, 17:40
Posts: 168
Hi

It would be pretty dumb at my part still I am unable to reproduce the bug, I mean can you explain how you are aggregating the files when it shows wrong output.

Regards
Ghamsham


Top
 Profile  
 
 Post subject: Re: differences in behavior, aggregated vs single file
PostPosted: 02 Dec 2016, 14:00 
User avatar

Joined: 15 Jan 2009, 22:34
Posts: 408
Hi Ghansham-

Load the grids through the General Files chooser. At the top choose Data Type of "Aggregate Grids by Time", select all four files, and Add Source. This will make one Data Source in the field selector. Display the Anvil Cloud Detection with color shaded plan view and that should show the error.

Thanks, Jay


Top
 Profile  
 
 Post subject: Re: differences in behavior, aggregated vs single file
PostPosted: 06 Dec 2016, 18:01 
User avatar

Joined: 15 Nov 2010, 17:57
Posts: 917
Hi Kris -

We merged Unidata's changes into McIDAS-V, and if you pick up today's (12/6) 1.7 beta1 nightly you should see that the aggregated behavior of the 'Anvil Cloud Detection ID' is now behaving better. All of the data values probe and plot, not just those less than 127, so there is no need to use the plugin I posted earlier. If you give this a look and notice any problems, please let us know.

Thanks -
Bob


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 17 posts ]  Go to page Previous  1, 2

All times are UTC


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group