NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

Re: [netcdf-java] Opening a GridDataset for large NcML aggregations

Aggregations are done when NetcdfDataset is opened, which should get cached. But I remember noticing that the aggregation was not getting cached. I will check it out. 

Jon Blower wrote:

Hi John,

Yes, I thought it might be the scanning that takes the time.  I had
assumed that the scanning takes place when you open a NetcdfDataset,
but I guess I'm wrong?  Perhaps this implies that I need a
GridDatatypeCache as well as a NetcdfDatasetCache?  (Caching is making
my head hurt, there are too many in my application already ;-)

Cheers, Jon

On Tue, Mar 18, 2008 at 4:22 PM, John Caron <caron@xxxxxxxxxxxxxxxx> wrote:

that should all work. the time may be in the scanning of 1000 files. Still, I 
have had other reports
 of extra slowness,  I will investigate when I can.



 Jon Blower wrote:
 > Hi John,
 >
 > It's a JoinNew, because the data files do not contain a time axis.
 > Hence we use a <scan> element and the time axis values are deduced
 > from the file names using dateFormatMark.  Our ncml file is attached.
 >
 > Yes, this is in our own application (ncWMS) not the TDS.
 >
 > Cheers, Jon
 >
 > On Tue, Mar 18, 2008 at 2:21 AM, John Caron <caron@xxxxxxxxxxxxxxxx> wrote:
 >> HiJon:
 >>
 >>  Are these JoinExisting or JoinNew ? How is the coordinate value specified? 
You are doing this in your own application, not in TDS?
 >>
 >>
 >>  Jon Blower wrote:
 >>  > Hi all,
 >>  >
 >>  > Using the Java NetCDF libraries, version 2.2.22, we have notice that
 >>  > opening a GridDataset using TypedDatasetFactory.open() is slow when
 >>  > the dataset in question is an NcML aggregation over a long timeseries
 >>  > (around a thousand timesteps, with one timestep per file).  The call
 >>  > to TDF.open() takes around 3 seconds every time, which is a problem
 >>  > for our application, where performance is important.
 >>  >
 >>  > We are using the NetcdfDatasetCache, which doesn't seem to help with
 >>  > this problem.  Is there something we can do to speed up the process of
 >>  > creating a GridDataset?  I don't think this was a problem for us in
 >>  > previous library versions, where we used a different syntax
 >>  > ("GridDataset gd = new GridDataset(nc);").
 >>  >
 >>  > I guess we could cache the GridDataset object in memory at the
 >>  > application level, but is there a reason not to do this (e.g. if this
 >>  > is a very large object)?
 >>  >
 >>  > Here's what we do using version 2.2.22:
 >>  >
 >>  >    NetcdfDataset nc = NetcdfDataset nc = 
NetcdfDatasetCache.acquire(filename);
 >>  >    GridDataset gd =
 >>  > (GridDataset)TypedDatasetFactory.open(DataType.GRID, nc, null, null);
 >>  >
 >>  > Thanks, Jon
 >>  >
 >>  _______________________________________________
 >>  netcdf-java mailing list
 >>  netcdf-java@xxxxxxxxxxxxxxxx
 >>  For list information or to unsubscribe, visit: 
http://www.unidata.ucar.edu/mailing_lists/
 >>
 >
 >
 >