NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

Re: [netcdf-java] Aggregating NetCDF files programmatically

On 12/11/2013 5:23 AM, Guy Griffiths wrote:

Hi all,

>
> I am looking into ways of aggregating NetCDF files programmatically
> for a new version of ncWMS.  The aim is to allow expressions such as
> "*.nc" to result in a single dataset, so long as all NetCDF files
> share the same variables/dimensions and differ only in the values on
> their time axes.
>
> I can do this by writing an NcML file, but this is a bit of a hack,
> and leaves files around.  We previously did this by manually keeping
> track of which time steps appeared in which files, but this doesn't
> easily fit with our new data model, and seems unnecessary when there
> are several Aggregation objects in the NetCDF-Java libraries which
> should be able to do the job for us.
>
> Searching lead me to this message on the list:
>
> http://www.unidata.ucar.edu/mailing_lists/archives/netcdf-java/2013/msg00029.html 
>
>
>
but unfortunately there were no responses. I was able to use the AggregationUtil class from that post to read the correct metadata, but like the person who posted I run into trouble when trying to actually read the data. I get a slightly different stack trace, which starts: 


> java.io.IOException: BAD: missing spi: analysed_sst at
> ucar.nc2.NetcdfFile.readData(NetcdfFile.java:1753) at
> ucar.nc2.Variable.reallyRead(Variable.java:850) ...
>
> and is then the same as in the other post.
>
> I would be very grateful if anyone could suggest a way of making
> aggregation work programatically, either with an adaptation of the
> above method or something completely new.
>
> Best Regards,
>
> Guy Griffiths

Hi Guy:
One workaround would be to construct the NcML as a string in memory and pass it to NcMLReader.readNcML () with a ByteArrayInputStream: 

   // read ncml through a InputStream.
    String ncml =
"<netcdf xmlns=\"http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2\";>\n" + 
        "<aggregation type=\"union\">\n" +
" <netcdf location=\"file:G:/work/jasmin/SVM01_npp_d20120120_t0531354_e0532596_b01189_c20120120120033631216_noaa_ops.h5\" />\n" + " <netcdf location=\"file:G:/work/jasmin/GMTCO_npp_d20120120_t0531354_e0532596_b01189_c20120120115420527613_noaa_ops.h5\" />\n" + 
        "  </aggregation>\n" +
        "</netcdf>";
NetcdfDataset aggregatedDataset = NcMLReader.readNcML( new ByteArrayInputStream(ncml.getBytes()), null ); 




  /**
   * Read NcML doc from an InputStream, and construct a NetcdfDataset.
   *
   * @param ins        the InputStream containing the NcML document
   * @param cancelTask allow user to cancel the task; may be null
   * @return the resulting NetcdfDataset
   * @throws IOException on read error, or bad referencedDatasetUri URI
   */
static public NetcdfDataset readNcML(InputStream ins, CancelTask cancelTask) throws IOException 

Also could construct a JDOM tree yourself and use:

  /**
   * Read NcML from a JDOM Document, and construct a NetcdfDataset.
   *
* @param ncmlLocation the URL location string of the NcML document, used to resolve reletive path of the referenced dataset, 
   *                     or may be just a unique name for caching purposes.
   * @param netcdfElem   the JDOM Document's root (netcdf) element
   * @param cancelTask   allow user to cancel the task; may be null
   * @return the resulting NetcdfDataset
   * @throws IOException on read error, or bad referencedDatasetUri URI
   */
static public NetcdfDataset readNcML(String ncmlLocation, Element netcdfElem, CancelTask cancelTask) throws IOException { 


John
  • 2013 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: