Re: in memory netcdf

• To: "ahoward" <ahoward@xxxxxxxxxxxx> <support-netcdf-java@xxxxxxxxxxxxxxxx>
• Subject: Re: in memory netcdf
• From: "John Caron" <caron@xxxxxxxxxxxxxxxx>
• Date: Mon, 8 Apr 2002 17:44:36 -0600

The latest release of netcdf java
(http://www.unidata.ucar.edu/packages/netcdf-java/) has a contributed class
InMemRandomAccessFile.java which does what you want. We havent finished
integrating it by adding a NetcdfFile constuctor for it, but that wouldnt be
hard (see public NetcdfFile(URL url)). I could add that if you are
interested, and willing to test it.

----- Original Message -----
From: "Russ Rew" <russ@xxxxxxxxxxxxxxxx>
To: "ahoward" <ahoward@xxxxxxxxxxxx>
Sent: Monday, April 01, 2002 4:47 PM
Subject: Re: in memory netcdf


> >To: russ@xxxxxxxxxxxxxxxx
> >From: ahoward <ahoward@xxxxxxxxxxxx>
> >Subject: Re: 20020327: in memory netcdf
> >Organization: NOAA/FSL
>
> Hi Ara,
>
> > don't know if you recall my name, but i was one of the early users of
your
> > netcdf java library a few years back.  in any case, i've switched
> > departments here at fsl and am now in the ITS group responsible for the
> > realtime data ingest.  we're currently in the process of revamping our
> > systems design and implementation language (c -->> c++).  i'm working on
> > generic handling of point data and need a good general data structure in
> > which to store all point data for eventual munging into output files of
> > either netcdf or grib format.  netcdf itself would be an ideal data
> > structure as the metadata contained in the file is complete enough for
my
> > purposes (with some creative uses of attributes) and the typed accessor
> > methods (c++ interface) are exactly the type of thing needed.
> >
> > my problem is that any use of netcdf requires a disk hit to read in an
> > existing cdf (output of ncgen) and each subsequent put/get also hits
disk.
> > i'm looking into replacing the IO layer (posixio.c) with a layer that
> > operates entirely in memory.  that would leave only the hit to read in
an
> > existing template (i'm also planning on looking at ncgen to see if it
> > could be modified to place a netcdf file structure in memory instead of
on
> > disk).  eg. i want to eliminate most, if not all, of the disk accesses
in
> > the c/c++ netcdf api.
> >
> > alternatively, applying the same set of requirements, and availibility
of
> > a constructor to the NcVar class might prove sufficient for our
purposes.
> >
> > my questions are :
> >
> > * do you know of any work along these lines, 3rd party or
> > otherwise?
>
> Andrew Janke <janke@xxxxxxxxxxx> proposed implementing something
> similar in the Java interface, which you can read about in this reply
> from John Caron to his question:
>
>   http://www.unidata.ucar.edu/cgi-bin/mfs/70/4439
>
> Glenn Davis also once implemented an experimental memory-mapped
> version of the i/o layer for netCDF 3.4 built on top of Unix mmap(2),
> which is described here:
>
>   http://www.unidata.ucar.edu/cgi-bin/mfs/70/2756
>
> and available in this file:
>
>   ftp://ftp.unidata.ucar.edu/pub/netcdf/contrib/mmapio.c
>
> that saves some disk I/O.
>
> > * am i completely off base in thinking the IO layer could be
> > replaced with an in memory IO layer
> >
> > any input appreciated, i was planning on cc'ing john caron on this as
well
> > but have lost his email.
>
> The I/O layer could be replaced with an in-memory layer.  But if you
> just want to access a small slice of data from a huge file, using
> random-access to read from the disk is cheaper than reading the whole
> file into memory first before you try to access the data slice from
> the in-memory copy.  But it sounds like that's not a concern for your
> application.
>
> By the way, Charlie Zender has developed a new C++ interface to
> netCDF, known as libnco_c++, which he describes in this netcdfgroup
> posting:
>
>   http://www.unidata.ucar.edu/glimpse/netcdfgroup-list/1484
>
> which emphasizes easy migration from the C interface ...
>
> --Russ
>