Page 1 of 1

Display many hours grib files no working

Posted: Tue Aug 04, 2020 10:51 pm
by Cloudy
When I upload multiple data files (grib files from https://nomads.ncep.noaa.gov/cgi-bin/fi ... s.20200804, for example) in mcidas-v V1.8, loading multiple files (for example: f00, f001, f002 ...) when creating display, only one hour appears instead of all the loaded ones. Clicking on Reload Data may cause the other hours to be loaded or not at random. Other times the load is not achieved after much insistence. The same happens if you click on Change Data, it may randomly work or not, generally not. Doing the same procedure with the same files in McIDAS-V Version: 1.0beta5 Build-date: 2009-10-28 19:21, it works fine.
I use Windows 10 Pro, 64-bit, Intel (R) Core (TM) 2 Quad CPU Q8300 @ 2.50 GHz, RAM 8.00 GB. I doing something wrong, or is it a bug?
Thank you very much

Re: Display many hours grib files no working

Posted: Wed Aug 05, 2020 9:18 pm
by jayh
Hi-

Bob found that there actually is an inquiry written up for this problem, inquiry 1676. These files have different "time" dimensions dependent on the forecast hour. Aggregate Grids By Time only looks for the "time" dimension, so any variables that use "time1" or "time2" don't aggregate. You can work around this by using a plugin (ftp://ftp.ssec.wisc.edu/pub/incoming/time1.jar) to get McV to aggregate with the time1 dimension instead of time, but there are other issues that variable names actually change dependent on the timesteps.

Here is Bob's thorough description of this issue:
Different fhour files have different time parameters. For example:

gfs.t06z.pgrb2.0p25.f000 - Only time dimension is "time"
gfs.t06z.pgrb2.0p25.f001 - Time dimensions of "time" and "time1"
gfs.t06z.pgrb2.0p25.f007 - Time dimensions of "time", "time1", and "time2"

The "Aggregate Grids By Time" data type only looks for the dimension "time". Once you get past the initialization (*.f000) many variables use other time dimensions meaning that they won't be aggregated. So, if you click on one of the variables that uses "time1" in the shape (can be seen in ncdump output) then you only get one timestep. You can get McV to use "time1" as the time dimension by using the time1.jar plugin, but you'll be limited by what times you add. For example, if you install the plugin and restart McV, you can choose the "Aggregate Grids by Time with time1" Data Type and select the *.f001, *.f002, *.f003, and *.foo4 files and click Add Source. You'll see that many of the variables that include "time1" in their shape can now be aggregated together. However, if you extend out to *.f007, most of the variables won't be able to be aggregated as some variables switch from using time1 to time2. Note that you can't use "Aggregate Grids by Time with time1" if you include the *.f000 file since time1 isn't a dimension in that file.

Since the 2009 version of McV there have been many updates, including changes to the netCDF version packaged in McV. Somewhere along the line, things in this package likely became more strict on the time dimension for variable aggregation.

Another issue is that the variable names that use the "time1" and "time2" dimensions actually change names dependent on the forecast hour. For example:

gfs.t06z.pgrb2.0p25.f001 - Total cloud cover (1_Hour Average) @ Entire atmosphere layer
gfs.t06z.pgrb2.0p25.f002 - Total cloud cover (2_Hour Average) @ Entire atmosphere layer
gfs.t06z.pgrb2.0p25.f003 - Total cloud cover (3_Hour Average) @ Entire atmosphere layer
gfs.t06z.pgrb2.0p25.f006 - Total cloud cover (6_Hour Average) @ Entire atmosphere layer
gfs.t06z.pgrb2.0p25.f007 - Total cloud cover (1_Hour Average) @ Entire atmosphere layer
gfs.t06z.pgrb2.0p25.f050 - Total cloud cover (2_Hour Average) @ Entire atmosphere layer
gfs.t06z.pgrb2.0p25.f354 - Total cloud cover (6_Hour Average) @ Entire atmosphere layer

If you try aggregating these files together with time1.jar (or a time2.jar file that changes the time dimension to "time2") it seems like the variable names are pulled from the first file selected. So, if you choose the *.f001 file as the first file of the aggregation, the *1_Hour_Average fields will list. You can create a loop of all of the timesteps, but only the timesteps that actually use 1_Hour_Average will be displayed. I don't know of a workaround for this. I've seen some GFS CONUS 80km grids don't specify the hour number in the variable name and just use "Mixed_Intervals" or something like that so the variable names are consistent throughout the run.


As for the reloading issue where the times of the variables change in the Field Selector, we think it's still all related to the different time variables. Bob loaded a group of 8 files with Aggregate Grids By Time originally only the variables that use the "time" dimension list multiple times available in the Field Selector. If you have one of these variables that uses "time" selected and reload the data, things don't change. If you select one of the variables that uses "time1" and then reload, things reload to load the data similar to the "time1" plugin where the variables that use "time1" as the dimension can be looped.

Besides the jar files, there is another solution that might help. Bob verified a workaround of using the Unix "cat" command (from inquiry 1676) still seems to work. He downloaded 6 gridded files and put them in the same directory. From a Terminal, he cd'd to that directory and ran:

cat * > gfs.grb

This gfs.grb file can then be loaded through the Gen>Files/Dir chooser, and Bob found that all of the variables aggregate together as expected. We are hoping either the jar files or using the "cat" command will provide a solution to get your desired loops. If you have any questions or comments, please let us know

Thanks, Jay