McIDAS-V Support Forums

Support and discussion of the McIDAS-V software package
Forum HomeMcIDAS-V Home
It is currently 04 Aug 2020, 02:13

All times are UTC




Post new topic Reply to topic  [ 17 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: differences in behavior, aggregated vs single file
PostPosted: 05 May 2016, 13:32 
User avatar

Joined: 28 Jan 2009, 19:27
Posts: 428
I am having a strange problem. I put 4 sample files on your ftp server at pub/incoming/A2015145.210[0-3].nc. See attached graphic. When I aggregate these four files and display the anvil cloud detection ID field, I get the display on the left. All values greater than 127 are set to missing, even though there are many values above 127. If I were to just load a single image via data type Grid Files (no aggregation), and then display the data, the values greater than 127 appear just fine. Any idea what the problem would be and how I can fix this hopefully without having to regenerate the files?


Attachments:
Screen Shot 2016-05-05 at 9.27.20 AM.png
Screen Shot 2016-05-05 at 9.27.20 AM.png [ 88.86 KiB | Viewed 289 times ]
Top
 Profile  
 
 Post subject: Re: differences in behavior, aggregated vs single file
PostPosted: 05 May 2016, 21:53 
User avatar

Joined: 15 Jan 2009, 22:34
Posts: 408
Hi Kris-

I have written up inq. Inq. 2342. for this problem. I have verified that the problem occurs in the nightly and version 1.5. I also have tested the IDV nightly which also shows the problem, so this information will be sent to the Unidata programmers.

Bob came up with a couple of workarounds to get your loop displaying correctly. First is a small script that displayed the four time steps of the files you uploaded:

Code:
# import  glob module
from glob import glob

# generate list of files
datapath = '/Users/rcarp/Data/Bedka/'
fileList = glob(datapath+'*.nc')

# define the loop
loop=[]

# loop through files in fileList and append together into the loop
for filename in fileList:
    params = dict(
        filename=filename,
        field='ir_anvil_detection',
        )

    data=loadGrid(**params)
    loop.append(data)

# create a display of the loop in the currently active window
layer=activeDisplay().createLayer('Color-Shaded Plan View', loop)


That method should work better with many files compared to the following suggestion.

You can also just load each individual file, then with a formula, display them in a loop.
1. Add Source for each netCDF file individually.
2. In the Field Selector under Formulas, choose Miscellaneous -> Make a time sequence from single time grids/images, then choose your Display type and click Create Display.
3. In the Field Selector window that pops up for the formula, you can choose the "Anvil Cloud Detection ID" for a time step, Click on Add Selected so it's populated in the Times field, and repeat this for each time. When you are done, click OK.

Hopefully one of these workarounds will help you currently. It's an odd problem since some of the variables display with the correct ranges. If you have any further questions or comments, just let us know.

Thanks, Jay


Top
 Profile  
 
 Post subject: Re: differences in behavior, aggregated vs single file
PostPosted: 05 May 2016, 23:47 
User avatar

Joined: 28 Jan 2009, 19:27
Posts: 428
Thanks Jay and team for looking into this. Unfortunately I have 104 files that I wanted to aggregate and display as a contour plan view in an animated 4 panel, with the contour display being overlaid on an IR satellite image. So a much more complicated situation than the script supports. I also recall that I would have to run this field through another formula that Bob had provided for dealing with a situation where 0 is a missing value, not a valid data value like many other arrays might have. See the following: viewtopic.php?f=14&t=1656&start=10

I seems as though I will need to wait for the Unidata folks to address this given the complication in trying to get this to work with my desired display.


Top
 Profile  
 
 Post subject: Re: differences in behavior, aggregated vs single file
PostPosted: 06 May 2016, 15:01 
User avatar

Joined: 15 Jan 2009, 22:34
Posts: 408
Hi Kris-

I talked with Bob, and if you are doing most of this work interactively, we were hoping you can incorporate this script into your interactive work.

1. Create your 4 panel display
2. Display your satellite data into the desired panel.
3. With that panel highlighted as the active display, then run the script from the jython shell that displays the contoured Anvil Cloud Detection ID. The last line of the script (the 'active.Display()') uses the highlighted active panel to display the data.

Would this work, or are we simplifying the situation too much?

Thanks, Jay


Top
 Profile  
 
 Post subject: Re: differences in behavior, aggregated vs single file
PostPosted: 06 May 2016, 21:56 
User avatar

Joined: 15 Jan 2009, 22:34
Posts: 408
Hi Kris-

With help from Unidata we discovered that the problem is that the anvil detection variable is declared as byte array, so, the valid range is -128 to 127. Somehow this prevents the valid range from being counted in the aggregation by time data source type.

For a workaround, Bob created a NcML wrapper file that aggregates the individual NetCDF files together and at the same time applies a scale_factor attribute to the ir_anvil_detection variable:

Code:
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
  <aggregation dimName="time" type="joinExisting">
    <variable name="ir_anvil_detection">
      <attribute name="scale_factor" type="short" value="1"/>
    </variable>
    <scan location="C:/Users/rcarp/Data/Bedka/" suffix=".nc" subdirs="false"/>
  </aggregation>
</netcdf>


You can change the "scan location" to point at the directory containing your files and load this NcML file through the General>Files/Directories chooser. Just choose the ncml file from the chooser instead of the netcdf files with Aggregate Grids by time process.

Let us know if this method works for you.

Thanks, Jay


Top
 Profile  
 
 Post subject: Re: differences in behavior, aggregated vs single file
PostPosted: 11 May 2016, 16:03 
User avatar

Joined: 28 Jan 2009, 19:27
Posts: 428
It was actually defined as an unsigned byte which has a range of 0 to 255. So if the Unidata reader can handle unsigned byte's, then all should be well. I'll give your wrapper a try.


Top
 Profile  
 
 Post subject: Re: differences in behavior, aggregated vs single file
PostPosted: 18 Oct 2016, 14:19 
User avatar

Joined: 04 Jan 2013, 20:53
Posts: 46
I have continued to find issues with display of NetCDF unsigned byte arrays where values greater than 128 are not appearing in my display. This is very problematic as many products I'm creating have unsigned bytes to minimize storage requirement but when I display, I see gaps in the image as shown in the attachment with the first post on this thread. Has Unidata looked into their treatment of byte arrays within IDV?


Top
 Profile  
 
 Post subject: Re: differences in behavior, aggregated vs single file
PostPosted: 29 Nov 2016, 14:51 
User avatar

Joined: 28 Jan 2009, 19:27
Posts: 428
Is there any action planned in the near future to address this issue related to display if unsigned byte data type? The data I'm working with has a lot of arrays of this type and I haven't been able to create the displays I really need with Mc-V. In this case, it isn't a "nice to have" request but rather a showstopper that inhibits me from analyzing/displaying my data properly.


Top
 Profile  
 
 Post subject: Re: differences in behavior, aggregated vs single file
PostPosted: 29 Nov 2016, 16:39 
User avatar

Joined: 11 Nov 2010, 17:40
Posts: 168
Hi sir

Can you repost a few files once to have a look. Long time back (6 years) when I was working with Tommy Jasmin Sir, we observed this issue and we found the IDV class ucar.unidata.data.DataUtil class toFloatArray(Array) method. We fixed it and sent it to IDV. It was fixed at that time. Let us see where it is not failing.
Regards
Ghansham


Top
 Profile  
 
 Post subject: Re: differences in behavior, aggregated vs single file
PostPosted: 29 Nov 2016, 16:52 
User avatar

Joined: 28 Jan 2009, 19:27
Posts: 428
I put some samples at:

ftp.ssec.wisc.edu
pub/incoming/A2015145.*.nc

Try aggregating these files and display Anvil Cloud Detection ID. You should see that values greater than 128 are missing


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 17 posts ]  Go to page 1, 2  Next

All times are UTC


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group