Page 1 of 2

differences in behavior, aggregated vs single file

Posted: Thu May 05, 2016 1:32 pm
by kbedka1
I am having a strange problem. I put 4 sample files on your ftp server at pub/incoming/A2015145.210[0-3].nc. See attached graphic. When I aggregate these four files and display the anvil cloud detection ID field, I get the display on the left. All values greater than 127 are set to missing, even though there are many values above 127. If I were to just load a single image via data type Grid Files (no aggregation), and then display the data, the values greater than 127 appear just fine. Any idea what the problem would be and how I can fix this hopefully without having to regenerate the files?

Re: differences in behavior, aggregated vs single file

Posted: Thu May 05, 2016 9:53 pm
by jayh
Hi Kris-

I have written up inq. Inq. 2342. for this problem. I have verified that the problem occurs in the nightly and version 1.5. I also have tested the IDV nightly which also shows the problem, so this information will be sent to the Unidata programmers.

Bob came up with a couple of workarounds to get your loop displaying correctly. First is a small script that displayed the four time steps of the files you uploaded:

Code: Select all

# import  glob module
from glob import glob

# generate list of files
datapath = '/Users/rcarp/Data/Bedka/'
fileList = glob(datapath+'*.nc')

# define the loop
loop=[]

# loop through files in fileList and append together into the loop
for filename in fileList:
    params = dict(
        filename=filename,
        field='ir_anvil_detection',
        )

    data=loadGrid(**params)
    loop.append(data)

# create a display of the loop in the currently active window
layer=activeDisplay().createLayer('Color-Shaded Plan View', loop)


That method should work better with many files compared to the following suggestion.

You can also just load each individual file, then with a formula, display them in a loop.
1. Add Source for each netCDF file individually.
2. In the Field Selector under Formulas, choose Miscellaneous -> Make a time sequence from single time grids/images, then choose your Display type and click Create Display.
3. In the Field Selector window that pops up for the formula, you can choose the "Anvil Cloud Detection ID" for a time step, Click on Add Selected so it's populated in the Times field, and repeat this for each time. When you are done, click OK.

Hopefully one of these workarounds will help you currently. It's an odd problem since some of the variables display with the correct ranges. If you have any further questions or comments, just let us know.

Thanks, Jay

Re: differences in behavior, aggregated vs single file

Posted: Thu May 05, 2016 11:47 pm
by kbedka1
Thanks Jay and team for looking into this. Unfortunately I have 104 files that I wanted to aggregate and display as a contour plan view in an animated 4 panel, with the contour display being overlaid on an IR satellite image. So a much more complicated situation than the script supports. I also recall that I would have to run this field through another formula that Bob had provided for dealing with a situation where 0 is a missing value, not a valid data value like many other arrays might have. See the following: viewtopic.php?f=14&t=1656&start=10

I seems as though I will need to wait for the Unidata folks to address this given the complication in trying to get this to work with my desired display.

Re: differences in behavior, aggregated vs single file

Posted: Fri May 06, 2016 3:01 pm
by jayh
Hi Kris-

I talked with Bob, and if you are doing most of this work interactively, we were hoping you can incorporate this script into your interactive work.

1. Create your 4 panel display
2. Display your satellite data into the desired panel.
3. With that panel highlighted as the active display, then run the script from the jython shell that displays the contoured Anvil Cloud Detection ID. The last line of the script (the 'active.Display()') uses the highlighted active panel to display the data.

Would this work, or are we simplifying the situation too much?

Thanks, Jay

Re: differences in behavior, aggregated vs single file

Posted: Fri May 06, 2016 9:56 pm
by jayh
Hi Kris-

With help from Unidata we discovered that the problem is that the anvil detection variable is declared as byte array, so, the valid range is -128 to 127. Somehow this prevents the valid range from being counted in the aggregation by time data source type.

For a workaround, Bob created a NcML wrapper file that aggregates the individual NetCDF files together and at the same time applies a scale_factor attribute to the ir_anvil_detection variable:

Code: Select all

<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
  <aggregation dimName="time" type="joinExisting">
    <variable name="ir_anvil_detection">
      <attribute name="scale_factor" type="short" value="1"/>
    </variable>
    <scan location="C:/Users/rcarp/Data/Bedka/" suffix=".nc" subdirs="false"/>
  </aggregation>
</netcdf>


You can change the "scan location" to point at the directory containing your files and load this NcML file through the General>Files/Directories chooser. Just choose the ncml file from the chooser instead of the netcdf files with Aggregate Grids by time process.

Let us know if this method works for you.

Thanks, Jay

Re: differences in behavior, aggregated vs single file

Posted: Wed May 11, 2016 4:03 pm
by kbedka1
It was actually defined as an unsigned byte which has a range of 0 to 255. So if the Unidata reader can handle unsigned byte's, then all should be well. I'll give your wrapper a try.

Re: differences in behavior, aggregated vs single file

Posted: Tue Oct 18, 2016 2:19 pm
by kbedka
I have continued to find issues with display of NetCDF unsigned byte arrays where values greater than 128 are not appearing in my display. This is very problematic as many products I'm creating have unsigned bytes to minimize storage requirement but when I display, I see gaps in the image as shown in the attachment with the first post on this thread. Has Unidata looked into their treatment of byte arrays within IDV?

Re: differences in behavior, aggregated vs single file

Posted: Tue Nov 29, 2016 2:51 pm
by kbedka1
Is there any action planned in the near future to address this issue related to display if unsigned byte data type? The data I'm working with has a lot of arrays of this type and I haven't been able to create the displays I really need with Mc-V. In this case, it isn't a "nice to have" request but rather a showstopper that inhibits me from analyzing/displaying my data properly.

Re: differences in behavior, aggregated vs single file

Posted: Tue Nov 29, 2016 4:39 pm
by ghansham
Hi sir

Can you repost a few files once to have a look. Long time back (6 years) when I was working with Tommy Jasmin Sir, we observed this issue and we found the IDV class ucar.unidata.data.DataUtil class toFloatArray(Array) method. We fixed it and sent it to IDV. It was fixed at that time. Let us see where it is not failing.
Regards
Ghansham

Re: differences in behavior, aggregated vs single file

Posted: Tue Nov 29, 2016 4:52 pm
by kbedka1
I put some samples at:

ftp.ssec.wisc.edu
pub/incoming/A2015145.*.nc

Try aggregating these files and display Anvil Cloud Detection ID. You should see that values greater than 128 are missing