NetCDF file readable by McIdas-V

[jinlee]

Hello,

I have created a NetCDF file with the following structure,

> ncdump 20160528053000-P1S-ABOM_OBS_B01-PRJ_GEOS141_2000-HIMAWARI8-AHI.cf.nc

netcdf \20160528053000-P1S-ABOM_OBS_B01-PRJ_GEOS141_2000-HIMAWARI8-AHI.cf {
dimensions:
time = 1 ;
record = 100 ;
variables:
int time(time) ;
time:units = "seconds since 1970-01-01 00:00:00" ;
time:long_name = "time" ;
float longitude(time, record) ;
longitude:units = "degrees_east" ;
longitude:long_name = "longitude (degrees)" ;
float latitude(time, record) ;
latitude:units = "degrees_north" ;
latitude:long_name = "latitude (degrees)" ;
float channel_0001_scaled_radiance(time, record) ;
channel_0001_scaled_radiance:units = "1" ;
channel_0001_scaled_radiance:long_name = "channel_0001_scaled_radiance" ;
data:

time = 1464413400 ;

longitude =
133.0603, 133.276, 133.4651, 133.641, 133.8085, 133.9699, 134.1266,
...
...


Because all the pixels are assumed to be at the same time I have created a dimension variable, "time" of size 1. Other variables are indexed by "time" and "record".

McIdas-V when reading the NetCDF file doesn't complain but when it comes to plotting point data it doesn't plot all the points. Can anyone able to point out any problem with the NetCDF file that I created? I have attached a cut down sample NetCDF file.

Thank you in advance.



Regards,

Jin

[bobc]

Hello Jin -

Would you be able to send us the original file (not cut down)? With the file you posted, I'm able to create a Value Plots display of the data, but actual contoured displays aren't working. Just to verify that the contoured displays aren't working because there's only one line of data in the display, looking at a larger version of the file may be helpful.

If it is too large to attach to the forum, you can post it to our anonymous ftp. Here are commands you can run :

ftp ftp.ssec.wisc.edu
user: anonymous
password: your email address
bin
cd pub/incoming
put filename
bye

Once you post the file, please let me know the file's name and I'll take a look.

Thanks -
Bob Carp
McIDAS Help Desk

[jinlee]

Hi Bob,

I have put my original NetCDF file on the ftp server you nominated,

/pub/incoming/20160528053000-P1S-ABOM_OBS_B01-PRJ_GEOS141_2000-HIMAWARI8-AHI.cf.nc


By the way when I try to do "Point Data Plot -> Create Display" using my NetCDF file I get following error message,

loading data
Value -292275055 for year must be in the range [-292275054,292278993]


Thank you very much for looking into this.


Regards,

Jin

[bobc]

Hi Jin -

I just wanted to let you know that I picked up your file and I'm looking into getting the data to display. You mentioned the Point Data Plot display. Are you interested in displaying this data as points, or as an image (e.g. Image Display, Contour Plan View)?

Thanks -
Bob

[jinlee]

Hi Bob,

For "Data Type" I chose "netCDF/GEMPAK Point Data files" when I was loading my data file. After "Add Source" completed "Field Selector" -> "Fields" only showed "Gridded Fields", "Gridded Fields (with first guess) and "Point Data" as available. It didn't offer "Image Display" so that's why I chose "Point Data". The actual data to be displayed is "scaled radiance" which probably is linearly related to radiance (W/m2). I also have other NetCDF files which contain brightness temperatures. Whatever display - Image Display or Contour Plan View - that works would be fine.



Regards,

Jin

[bobc]

Hi Jin -

I had been originally been trying to load this data as a gridded data source, with the "Grid files (netCDF/GRIB/OPeNDAP/GEMPAK)" Data Type. In order to get this working, I had to write up a NcML file that defined "1.0E30" as a missing value for latitude and longitude. I also set a missing value of "9.969209968386869E36" for channel_0001_scaled_radiance. With the NcML file in the same directory as the full-sized netCDF file, I was able to load this NcML file as a grid:

20160528053000-P1S-ABOM_OBS_B01-PRJ_GEOS141_2000-HIMAWARI8-AHI.cf.ncml

(636 Bytes) Downloaded 394 times

Note that you can change location= in the NcML file to specify the full path to the original netCDF file if you want the NcML and netCDF files in separate directories.

With the data source from the NcML file, there are a couple things that I'm able to do:

1. With the "Plan Views > Value Plots" display type, I can plot the data points/values at their correct locations. For example:
   

   

   Value Plots display

2. With the "Plan Views > Color-Shaded Plan View" display type, I can get the data to display as long as I set the Display Mode in the Layer Controls to "Mesh". For example:
   

   

   Color-Shaded Plan View, mesh mode

I'm still at the point where the Image Display and other Plan View displays don't work correctly. I'm going to look into this some more, and likely ask some programmers for their thoughts. I believe there's a possibility of a few different things:

1. This could be a bug in McIDAS-V, where the Image and Plan View displays should work with this NcML file as-is.
2. There are additional modifications that can be made to the NcML to get the Image and Plan View displays working.
3. There may be an issue with the file using the "record" dimension. Other netCDF files that I've been able to work with in McIDAS-V in the past haven't had this "record" dimension, but rather "x/y" or "line/element".

I'll keep you updated on any progress with this.

Thanks -
Bob

[jinlee]

Hi Bob,

I loaded the data as gridded data and used your NcML, and this allowed me to do Value Plots. However, "Plan Views > Color-Shaded Plan View" display doesn't seem to work for me even when I choose "Layer Controls > Display Mode" to be "Mesh".

Thank you very much for looking in this.



Regards,

Jin

[bobc]

Hi Jin -

I looked into this a bit and spoke with a programmer, and we believe the issue is with the "record" dimension in the file. Would it be possible for you to re-create these files with x/y dimensions instead of record?

From the programmer:

The file is structured as point data. Since 30250000 = 5500x5500 you might be able to redefine (time, record) as (time, y=5500, x=5500) via NcML.

I looked into this with NcML. It looks like you can add/remove dimensions fairly easily. However, it doesn't appear to be possible to actually change the shape of the variables. For example, the shape of channel_0001_scaled_radiance would have to change from "(time=1, record=30250000)" to "(time=1, x=5500, y=5500)". Also, from the programmer's response, it appears like you were on the right track attempting to load this data as a NetCDF point data source instead of a grid.

If you are able to re-create these files with x/y dimensions, please let me know if they successfully load and display.

Thanks -
Bob

[jinlee]

Hi Bob,

Sorry I haven't been able to reply sooner. I thought I would get to the bottom of this without bothering you again but I think I still have a problem with my NetCDF file.

Following your suggestion I changed the shape of my variables as well as replacing the "record" dimension with "x" and "y". Here's the structure of my new NetCDF file,

netcdf \20160528053000-P1S-ABOM_OBS_B01-PRJ_GEOS141_2000-HIMAWARI8-AHI.cf {
dimensions:
time = 1 ;
x = 5500 ;
y = 5500 ;
variables:
int time(time) ;
time:units = "seconds since 1970-01-01 00:00:00" ;
time:long_name = "time" ;
float longitude(time, x, y) ;
longitude:units = "degrees_east" ;
longitude:long_name = "longitude (degrees)" ;
longitude:missing_value = 1.e+30f ;
float latitude(time, x, y) ;
latitude:units = "degrees_north" ;
latitude:long_name = "latitude (degrees)" ;
latitude:missing_value = 1.e+30f ;
float channel_0001_scaled_radiance(time, x, y) ;
channel_0001_scaled_radiance:_FillValue = 1.e+20f ;
channel_0001_scaled_radiance:units = "1" ;
channel_0001_scaled_radiance:long_name = "channel_0001_scaled_radiance" ;


I have also followed the example .ncml file you sent me and used "missing_value" and "_FillValue" attributes for variables. All these changes however doesn't seem to have made the file readable by McIdas-V. Do you mind having a look at the file and suggest what I need to do to allow McIdas-V to read it? I uploaded a copy of my file to the following location on the SSEC ftp server,

ftp.ssec.wisc.edu:/pub/incoming/20160528053000-P1S-ABOM_OBS_B01-PRJ_GEOS141_2000-HIMAWARI8-AHI.cf.nc


Thank you very much for your help.


Regards,

Jin

[bobc]

Hi Jin -

Apologies on backtracking a bit, but I dug around and actually found some point netCDF atmospheric motion vectors from Himawari 8 that were given to us from the Australian Bureau of Meteorology. These files load in as a netCDF point data source, and you can display the data as either points (using the point data field) or as an image (using one of the gridded fields). Where the backtracking comes into play is that these files actually do use a "record" dimension, not x and y.

Here's some ncdump output of a file:

Code: Select all

netcdf V:/rcarp/Data/NetCDF/JohnLeMarshall/new_amv.HIMAWARI-8.2015119.0000.ch_14.nc {
  dimensions:
    record = 33274;
  variables:
    int Time(record=33274);
      :units = "seconds since 1970-01-01 00:00:00";
      :long_name = "date/time of measurement";

    float Altitude(record=33274);
      :units = "m";
      :coordinates = "Time Latitude Longitude Altitude";
      :long_name = "Derived altitude of tracer from pressure (m)";

    float MedianPress(record=33274);
      :units = "hPa";
      :coordinates = "Time Latitude Longitude Altitude";
      :long_name = "Pressure assignment of tracer (hPa)";

    float Wind_Speed(record=33274);
      :units = "m.s-1";
      :coordinates = "Time Latitude Longitude Altitude";
      :long_name = "Speed of average wind vector (m/s)";

    float Wind_Dir(record=33274);
      :units = "degrees";
      :coordinates = "Time Latitude Longitude Altitude";
      :long_name = "Direction of average wind vector (degrees)";

    float Latitude(record=33274);
      :units = "degrees_north";
      :long_name = "Wind Latitude  (degrees)";

    float Longitude(record=33274);
      :units = "degrees_east";
      :long_name = "Wind Longitude  (degrees)";

}

Note a couple key differences between your netCDF file and this one above that is working properly:

1. There is no "time" dimension in these files, but rather a "Time" variable. Even though time=1 isn't a dimension, all of the points have the same timestep.
2. For the variables that are intended to be plotted (e.g. "Wind_Speed") there is a coordinates attribute.

I'd suggest that you attempt modeling your netCDF files off this one. I put it on ftp at:
ftp://ftp.ssec.wisc.edu/pub/incoming/new_amv.HIMAWARI-8.2015119.0000.ch_14.nc

... this file should remain at that location for a week. If you need to grab it after it has been removed, let me know and I'll re-post it. Another note about this file is that it has a lot fewer data points than yours does (33,274 vs 30,250,000) and an Image Display of a field takes about 20 seconds to complete. To cut back on the time it takes for your data to display, you may want to consider displaying only a small geographical area of your data (via the Region subset tab in the Field Selector) or different sampling options in the Grid Parameters subset tab so not all points are used in the display.

Thanks -
Bob