NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

Re: netCDF library

You could also write your data in a sparse format similar to something like CSR, using netCDF files as a container. 

Rob

Russ Rew wrote:

Hi,

Dave Allured wrote:


... Consider these options to reduce archival file size:
1. Keep your current Netcdf format, but store your files gzip'ed. Make uncompressing a standard part of opening the file. Many application languages will allow you to call the shell to gunzip and delete a temporary file, so you can automate this. gunzip is rather fast, as I recall. As you stated, your file size is reduced by 99%.
2. Netcdf 16-bit packed format. Reduce file size by 50%. You get 16 bits for your combined precision and dynamic range.
3. Netcdf 8-bit packed format. Reduce file size by 75%. You get 8 bits for your combined precision and dynamic range. 

That's good advice, and here's two additional options:

4.  Switch to using HDF5, which supports a form of access called
    "chunking", a multidimensional tiling of arrays that permits
    accessing data a tile at a time, with each tile compressed
    separately.  Thus if you only access a small subset of the data in
    a large file, only a small part of the data is uncompressed, so
    you don't pay the penalty of uncompressing the whole file on open.
    With chunking, you can also choose which variables to compress
    instead of compressing the entire file.

5.  Wait for netCDF-4, which also supports chunking and compression by
    delegating it to the HDF5 storage layer.  Currently the release of
    netCDF-4 is being delayed until the corresponding HDF5 1.8
    software on which it depends is ready (estimated December 2006),
    but if you can wait, it may be easier to convert your code to use
    netCDF-4 than HDF5.  You can also test the use of chunking in the
    current netCDF-4 alpha release, but note that it uses artifacts
    that will not be supported in the real netCDF-4 software.

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
russ@xxxxxxxxxxxxxxxx                     http://www.unidata.ucar.edu

==============================================================================
To unsubscribe netcdfgroup, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================



==============================================================================
To unsubscribe netcdfgroup, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================
  • 2006 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: