Create time Series from date/time in filename (ncml wrapper)

Useful hints
User avatar
joleenf
Posts: 1123
Joined: Mon Jan 19, 2009 7:16 pm

Create time Series from date/time in filename (ncml wrapper)

Post by joleenf »

Thank-You to Tommy Jasmin for helping me with this ncml wrapper to create a time series from a group a files which only contain date/time in the filename. There is a detailed tutorial at UNIDATA: http://www.unidata.ucar.edu/software/ne ... ation.html

I thought it might be helpful for V users to have a snippet of code posted here as well.

I have series of netCDF with filenames of


geocatL2.GOES-13.2011221.173200.hdf.ci.hdf
geocatL2.GOES-13.2011221.174500.hdf.ci.hdf
geocatL2.GOES-13.2011221.181500.hdf.ci.hdf
geocatL2.GOES-13.2011221.183200.hdf.ci.hdf
geocatL2.GOES-13.2011221.184500.hdf.ci.hdf
geocatL2.GOES-13.2011221.190200.hdf.ci.hdf
geocatL2.GOES-13.2011221.191500.hdf.ci.hdf
geocatL2.GOES-13.2011221.193200.hdf.ci.hdf
geocatL2.GOES-13.2011221.194500.hdf.ci.hdf
geocatL2.GOES-13.2011221.200200.hdf.ci.hdf
geocatL2.GOES-13.2011221.201500.hdf.ci.hdf

The fields appear in the field selector as

Screen Shot 2012-02-10 at 9.48.42 AM.png


Date/time dimensions are not in the file, so the Time tab does not appear in the bottom right panel under displays.

To create a time sequence of the field box_average_11um_ctc, use a ncml wrapper. In this case, aggregate the desired field (<variableAgg name="box_average_11um_ctc"/>) together into a time series (<aggregation dimName="time" type="joinNew">) based on the new time coordinate variables assigned to the files (coordValue="yyyy-mm-dd HH:mmZ")

Code: Select all

<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
   <aggregation dimName="time" type="joinNew">
        <variableAgg name="box_average_11um_ctc"/>
        <netcdf location="geocatL2.GOES-13.2011221.173200.hdf.ci.hdf" coordValue="2011-08-09 17:32Z"/>
        <netcdf location="geocatL2.GOES-13.2011221.174500.hdf.ci.hdf" coordValue="2011-08-09 17:45Z"/>
        <netcdf location="geocatL2.GOES-13.2011221.181500.hdf.ci.hdf" coordValue="2011-08-09 18:15Z"/>
        <netcdf location="geocatL2.GOES-13.2011221.183200.hdf.ci.hdf" coordValue="2011-08-09 18:32Z"/>
        <netcdf location="geocatL2.GOES-13.2011221.184500.hdf.ci.hdf" coordValue="2011-08-09 18:45Z"/>
        <netcdf location="geocatL2.GOES-13.2011221.191200.hdf.ci.hdf" coordValue="2011-08-09 19:12Z"/>
        <netcdf location="geocatL2.GOES-13.2011221.193200.hdf.ci.hdf" coordValue="2011-08-09 19:32Z"/>
   </aggregation>
</netcdf>


LoopGeocat.ncml
(888 Bytes) Downloaded 535 times


Next, load the ncml wrapper (LoopGeocat.ncml) and select Aggregate Grids by Time as the data type.
Screen Shot 2012-02-10 at 9.47.55 AM.png

Notice that when "LoopGeocat.ncml" is selected as the Data Source, the new aggregated field appears at the end of the field list and has a time sequence associated with the data.
Screen Shot 2012-02-10 at 9.52.33 AM.png



Joleen
User avatar
joleenf
Posts: 1123
Joined: Mon Jan 19, 2009 7:16 pm

Re: Create time Series from date/time in filename (ncml wrapper)

Post by joleenf »

Since the filenames of these data files contain the date and time information, I was able to get this to work by using a date format string. This is a quicker alternative for these files

Code: Select all

<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
  <aggregation dimName="time" type="joinNew">
    <variableAgg name="box_average_11um_ctc"/>
    <scan location="/Users/joleenf/data/turb/20110809_aa734/" dateFormatMark="geocatL2.GOES-13.#yyyyDDD.HHmmss" suffix=".hdf" subdirs="false" />
  </aggregation>
</netcdf>


ciwrapper.ncml
(274 Bytes) Downloaded 527 times


The documentation for the ncml aggregation is at
http://www.unidata.ucar.edu/software/ne ... ation.html

Joleen
User avatar
joleenf
Posts: 1123
Joined: Mon Jan 19, 2009 7:16 pm

Re: Create time Series from date/time in filename (ncml wrapper)

Post by joleenf »

This is a feature request, but also a warning to those who wish to use this for polar orbiting data. When aggregating grids by time, it is assumed that the domain of the grids are the same. Therefore, the procedure listed above will not work for polar orbiting data, because the navigation is retained from the first file in the list and applied to all the grids. It would be ideal to read each grid's navigation independently.

Thanks,
Joleen
User avatar
bobc
Posts: 988
Joined: Mon Nov 15, 2010 5:57 pm

Re: Create time Series from date/time in filename (ncml wrapper)

Post by bobc »

Hi Joleen -

Thanks for the feature request. I have written up McIDAS-V Inquiry #1252 to cover the request of adding the ability for the 'Aggregate Grids by Time' data type to work properly with files whose navigation changes from time to time.

http://mcidas.ssec.wisc.edu/inquiry-v/?inquiry=1252

Thanks -
Bob Carp
User avatar
hproe
Posts: 504
Joined: Sat Nov 27, 2010 3:46 pm

Re: Create time Series from date/time in filename (ncml wrapper)

Post by hproe »

God peolpe -

I have tried to ncml-wrap some netCDF files holding Meteosat-4 data without date/time dimension. However, both using Joleen's example further up in this thread or the example 3 given in http://www.unidata.ucar.edu/software/thredds/v4.6/netcdf-java/ncml/Aggregation.html, I get puzzling results with a recent 1.6beta build. I have uploaded to the SSEC server the files for both instances like:
1. AggMy.7z
2. AggUnidata.7z

In bothe cases the wrapping does not work, but for different reasons:
1. Data (of first image) get loaded, but no time sequence is created
2. Add Source ends in Non-Compliant NetCDF Tool

Any idea what goes wrong here?

cheers, HP
User avatar
bobc
Posts: 988
Joined: Mon Nov 15, 2010 5:57 pm

Re: Create time Series from date/time in filename (ncml wrapper)

Post by bobc »

Hi HP -

I'm a bit confused as to why your first example isn't working. One thing that looks to be incorrect in your myAggNew.ncml file is how you specified the variableAgg name. You used "MVIRI visible pixel counts", which is how the field lists in the Field Selector, but it should be how the field is actually specified in the file, which is "ch1". You can see this "ch1" if you mouse over the field in the Field Selector. This makes the myAggNew.ncml file:

Code: Select all

<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
  <aggregation dimName="time" type="joinNew">
   <variableAgg name="ch1"/>
   <netcdf location="VIS-IR-MET4_19920326080000.nc" coordValue="1996-03-26 08:00UTC"/>
   <netcdf location="VIS-IR-MET4_19920326083000.nc" coordValue="1996-03-26 08:30UTC"/>
  </aggregation>
</netcdf>

... however, when this file is loaded the "MVIRI visible pixel counts" field doesn't list in the Field Selector. I'll keep looking into this. However, the alternative method of using the dateFormatMark from the filenames does work well:

Code: Select all

<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
  <aggregation dimName="time" type="joinNew">
    <variableAgg name="ch1"/>
    <scan location="" dateFormatMark="VIS-IR-MET4_#yyyyMMddHHmmSS" suffix=".nc" subdirs="false"/>
  </aggregation>
</netcdf>

... this assumes that this NcML file is in the same directory as the NetCDF files since the "scan location" is set as an empty string, but a file path to the NetCDF files could be specified if this NcML file isn't in the same directory. When this NcML file is loaded in, all 3 fields list in the Field Selector, and only the "ch1" field can be displayed as a loop. Searching online, it looks like this is the most standard way of aggregating files with a time defined in the filename.

As for your second file, I'm replicating that this brings up the Non Compliant NetCDF editor in McIDAS-V. It looks like with this file there's no real time value specified, just numerical values of "0", "10", and "99". This file can be loaded in NetCDF's ToolsUI. If you keep following down Unidata's Aggregation page you linked to, it looks like more work is done on this file to get it to a point where real times are specified. Eventually, you get to this:

Code: Select all

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">

  <variable name="time" type="int">
    <attribute name="units" value="months since 2000-6-16 6:00"/>
    <attribute name="_CoordinateAxisType" value="Time"/>
    <values>0 1 2</values>
  </variable>

  <aggregation dimName="time" type="joinNew">
    <variableAgg name="T"/>
    <netcdf location="time0.nc"/>
    <netcdf location="time1.nc"/>
    <netcdf location="time2.nc"/>
  </aggregation>

</netcdf>

This can be added to and displayed in McIDAS-V as a loop. I'll let you know if I can find out why the first case of passing times through coordValue isn't working, but please let me know if the dateFormatMark option doesn't work with your data or if you have any other questions.

Thanks -
Bob Carp
User avatar
hproe
Posts: 504
Joined: Sat Nov 27, 2010 3:46 pm

Re: Create time Series from date/time in filename (ncml wrapper)

Post by hproe »

Hi Bob -

Many thanks for your help. The 'scan code' works like a charm. Next time I will read the Unidata page right to the end, sorry.

HP
User avatar
bobc
Posts: 988
Joined: Mon Nov 15, 2010 5:57 pm

Re: Create time Series from date/time in filename (ncml wrapper)

Post by bobc »

Hi HP -

I'm glad that the dateFormatMark option is working well. I'm still stumped on why the other method of supplying a coordValue time for each file isn't working. I think that it's because of the "_CoordinateAxisType" for time. With your original NcML file where each file is individually listed, looking at the metadata lists "String time(time=2)", but no attributes are listed below it. With the other NcML file where dateFormatMark is used, "String time(time=2)" has an attribute of "_CoordinateAxisType = "Time"". There are a few examples of setting "_CoordinateAxisType" on the Unidata NcML page, but none of them match what we are doing here, listing each file and supplying a time with coordValue.

I have other types of files from users in the past where I've been able to use your original method (list each file and specify date/time info via coordValue), but for some reason this isn't working here. I've written up Inquiry 2302 to investigate.

Thanks -
Bob
User avatar
hproe
Posts: 504
Joined: Sat Nov 27, 2010 3:46 pm

Re: Create time Series from date/time in filename (ncml wrapper)

Post by hproe »

Hi -

I have a similar case to mine above with MVIRI netCDF files. This time it is grib files with the SEVIRI MPE product. I tried to adapt the NcML file to the new filename template and to the changed variable name. But when loading the NcML I just get the 'Non-Compliant NetCDF Tool'. I have copied the variable name from over the field name. aggMPE.ncml and 2 sample grib files are uploaded as aggMPE.7z. I note that this time the field name is nested one level deeper than it was in the former case.

season's greetings, HP
User avatar
bobc
Posts: 988
Joined: Mon Nov 15, 2010 5:57 pm

Re: Create time Series from date/time in filename (ncml wrapper)

Post by bobc »

Hi HP -

Thanks for posting the grib files and your wrapper. Doing a ncdump on the grib files show that there are a couple groups contained within the file: "SpaceViewPerspective_3712X3712-3p323E-6N-p02695E" and "LatLon_24X24-31p30N-31p30E". The dimensions (x, y, time, lat, lon) are under these groups. The NetCDF library in McIDAS-V doesn't handle group names. In fact, even trying to use the "Aggregate Grids by Time" Data Type doesn't work with these files because McIDAS-V can't find the 'time' dimension contained within the groups.

We've had a user in the past try aggregating NetCDF files with groups and discovered it didn't work, so we wrote up Inquiry 1255, which was later closed since it was decided this would be handled in Inquiry 1252.

I've been doing some research online and trying different things with the NcML file but I've been unable to get anything to work as of yet. I'll follow up with you if I make any progress.

Thanks -
Bob
Post Reply