exportGridToNetcdf() with from lat-lon limted area

Post ideas for new functionality you'd like to see in McIDAS-V or ideas for new tutorials.
Post Reply
User avatar
hproe
Posts: 504
Joined: Sat Nov 27, 2010 3:46 pm

exportGridToNetcdf() with from lat-lon limted area

Post by hproe »

Hi -
In case of a global GFS data source exportGridToNetcdf() exports the selected meteorological fields of the global area. In order to limit the data volume and the time of file creation, I would like to be able to export only data from a lat-lon delimited region, in particular for one that straddles the 0° longitude. Is there already a procedure that makes this possible (my search was not successful)? Or, would this be a valid feature request?
cheers, HP
User avatar
bobc
Posts: 988
Joined: Mon Nov 15, 2010 5:57 pm

Re: exportGridToNetcdf() with from lat-lon limted area

Post by bobc »

Hi HP,

When using the McV formula to export a grid to netCDF the Field Selector window where you select the field you want to export should contain a Region tab where you can draw a bounding box around your area of interest. However, I have a feeling that you're running into the problem written up as Inquiry 1626, where you can't subset across a boundary in the Region tab of the Field Selector. With most global GFS data I've looked at, the Region tab has a boundary at 0 degrees longitude that you can't subset across.

One thing that may work is if we were to find a global grid at the same resolution of your data that had an edge at 180 degrees longitude (the dateline) instead of the prime meridian. If we found a grid like this, then it may be possible to use the Resample Grid formula to change your gridded data source to match the domain of the other data, which may allow for subsetting across 0 degrees longitude since it would no longer be at a boundary in the Region tab of the Field Selector.

What resolution is the data that you're working with? If you let me know this, I can try finding a grid of the same resolution with an edge at the dateline to try out my suggestion above.

Thanks,
Bob Carp
User avatar
hproe
Posts: 504
Joined: Sat Nov 27, 2010 3:46 pm

Re: exportGridToNetcdf() with from lat-lon limted area

Post by hproe »

Hi Bob -

I realize that I am hitting on a very old topic already discussed in archaic IDV times - the 0° longitude discontinuity that prevents selection/drawing of areas straddling this longitude.
With the cartopy/metpy packages charts straddling 0° may be drawn easily in python. The narrow gap along the 0° longitude remains though, but that is not much disturbing (see attached sample).
The problem I have is with the format of the model (GFS) files. For the time being metpy only accepts netCDF files. I have not found a free source that offers adequate recent analysis and forecast files in netCDF - all is in grib. McV offers a very straightforward way to extract data fields from grib to netCDF files: the 'Write Local Grid' command (right-clicking the grib file under the 'Data Sources' tab).
However, due to old problem mentioned above, I have to take the global data extent and cannot limit the data extraction to an area around 0° longitude. This, together with a hardwired(?) upper limit for the volume of the output file and in the case of the GFS forecast with all time steps, limits the extraction to the 1°-resolution and to 2-3 fields/levels, like shown in the sample chart.

Your idea, to look for a grid where the irregularity is shifted to 180°, has surfaced also in my mind, but I have never hit on something useful. I also add that up to now I have done the data extraction only from the GUI and have not tried to script the process.

All this, for the time being, boils down to two questions:
1. Can the file size of the netCDF file be increased?
2. The extraction is very slow (on my notebook at least). Why?

cheers, HP
Attachments
2018110506_ZT500hPa.png
User avatar
bobc
Posts: 988
Joined: Mon Nov 15, 2010 5:57 pm

Re: exportGridToNetcdf() with from lat-lon limted area

Post by bobc »

Hi HP,

A couple followups from earlier:
  1. I tried running loadGrid with the latLonBounds keyword where I specified a box that crossed the prime meridian. This did not work, as only data from one side of the prime meridian was included in the data source.
  2. I found an old global netCDF file that had a boundary at the dateline. I tried various ways of resampling the GFS data into the older netCDF file, but this didn't work either, as only GFS data from one side of the prime meridian was included in the data source.

As for your questions:
  1. I loaded a global 1 degree GFS file that included all fields, levels, and timesteps. Once I was in the Field Selector, I right-clicked on the Data Source and chose "Subset and Write Grid". After entering my filename, in the Select Fields window I clicked the "Select All" box and clicked OK. Once this netCDF file was done being created, I loaded it and looked at the Properties window of the data source. It seems like this netCDF file I wrote out is identical to the original GFS data from the catalog in that it has the same fields, times, levels, and resolution. Could you please give me a step-by-step of how you're running into the problem where the netCDF file you wrote doesn't include all of the data?
  2. The process from #1 above took about 4 minutes to complete. This seems somewhat reasonable since McV has to read/write all of the 2D and 3D (including all of the 3D levels) fields, but I'll check with a programmer to get their thoughts. Are you seeing this process take a lot longer than that?

Thanks,
Bob
User avatar
hproe
Posts: 504
Joined: Sat Nov 27, 2010 3:46 pm

Re: exportGridToNetcdf() with from lat-lon limted area

Post by hproe »

Hi Bob -

I have experimented a bit more with writing to netCDF. As regards your points 1&2 under 'as for your questions', the download (using the same procdure as you) of the full data set (from the 1-degree set at 'GFS One Degree - Global Coverage' takes hours (never wait for it to end). When I reduce the data set to the 5 fields I really need it takes about 65 minutes (identical times from 2 runs). The reduced data set is:
MSLP_Eta model_reduction_msl
u-componen_of_wind_isobaric (at 850hPa) - bare in mind all 31 levels will be downloaded for
v-componen_of_wind_isobaric (at 850hPa) - the 3D fields, though only 1 is used for each field
Gepotential_height_isobaric (at 500hPa)
Temperature_height_isobaric (at 500hPa)

When I download this reduced data set from 'GFS 1.0 Degree (NOAAPORT)' it takes just 10 minutes, which is acceptable. Consider that the NOAA Port set has only 12 levels and 41 time steps (and a reduced set of fields). For the time being the NOAA Port data are fine by me, though I miss the 3-hour time steps early in the forecast.

I have created a bundle for the download from NOAA Port of the field list specified above (NGFS1degSelection.mcv attached). However, it has 2 problems. It requires that the out put file pre-exists and that, more serious, it reverts to the data set used at bundle creation and not to the 'latest' data set, i.e. the bundle does not work. I note the following when comparing the XML code of NGFS1degSelection.mcv to code from bundles that simply display the most recent GFS fields. In the latter bundles I find the word 'latest' somewhere down the code lines, whereas NGFS1degSelection.mcv does not have it. Is the omission of referring to the most recent collection a bug?

I am not versed at all in the bundle XML, but I assume that for the time being the bundle could be tweaked to load the most recent data collection and not to require that the nc file pre-exists. This would get me going!

Of course my final wish still is to get the data collection at 0.25deg resolution for a small area straddling the 0deg longitude and being able to specify the levels and time step for the selected fields.

cheers, HP

PS I just found out that, unlike with the levels of 3D data, only pre-selected time steps get written to the netCDF file. In order to achieve this, one of the fields to be downloaded to netCDF has first to be displayed (after having selected the appropriate time steps for them). The remaining fields may then be ticked on the tables presented before writing the file
Attachments
NGFS1degSelection.mcv
(107.01 KiB) Downloaded 390 times
User avatar
bobc
Posts: 988
Joined: Mon Nov 15, 2010 5:57 pm

Re: exportGridToNetcdf() with from lat-lon limted area

Post by bobc »

Hi HP,

Apologies that I just now got a chance to look at this one. I'm replicating your findings that there is no mention of the word "Latest" in your bundle. I have a feeling that the bundle was created from an absolute time of a model run instead of "latest". I made 2 bundles, one with the "latest" run and another with the run from 11/15 at 0Z. I took the following lines from my "latest" bundle and moved them to my "11/15 0Z" bundle:

Code: Select all

                            <method name="put">
                                <string><![CDATA[RESOLVERURL]]></string>
                                <string><![CDATA[http://thredds.ucar.edu/thredds/catalog/grib/NCEP/GFS/Global_onedeg/latest.xml]]></string>
                            </method>

These lines were added after the:

Code: Select all

                    <property name="Properties">
                        <object class="java.util.Hashtable">

lines... Once this was done, loading my 11/15 0Z bundle now goes to the "Latest" model run. Perhaps you could give this a try with your bundle? I'd suggest adding these lines to your bundle file:

Code: Select all

                            <method name="put">
                                <string><![CDATA[RESOLVERURL]]></string>
                                <string><![CDATA[http://thredds.ucar.edu/thredds/catalog/grib/NCEP/GFS/Global_onedegree_noaaport/latest.xml]]></string>
                            </method>

right at line 51 and see if things work. You don't have to delete anything from your existent bundle. Note that you may need to adjust the indentation of the lines above to make things line up correctly in your bundle.

Thanks,
Bob
User avatar
hproe
Posts: 504
Joined: Sat Nov 27, 2010 3:46 pm

Re: exportGridToNetcdf() with from lat-lon limted area

Post by hproe »

Hi Bob -

Somewhat dirty, but it works!

BTW further up from where you inserted snippet, at line 14, there is already a reference to the nc file:

Code: Select all

<string><![CDATA[C:\Users\hproe_000\Desktop\NGFS1degSelection.nc]]></string>

Is this the reason why the bundle requests this file to exist, as I reported already earlier on?

Anyway, I think that for the time being I can now construct something useful.

HP
Post Reply