limits of netCDF library

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: limits of netCDF library
• From: Peter Halvorson <pjh@xxxxxxxxx>
• Date: Mon, 29 Apr 1996 16:56:09 PDT

The list has been kind of quiet, is it time to start the time debate again?
Just kidding.

I started looking at netCDF a month ago and was impressed by the quality
of the library, documentation, utilities, and post processing programs
(I've used xmgr without netCDF for years, GMT, ferret, and MESHTV all 
sound interesting).  The output of ncdump is a thing of beauty.  The
developers have been two steps ahead of me all the way.  But (there is
always a but), for the use I'm thinking of the single unlimited
dimension may not be sufficient.  I looked through the email group
archives and found a few people who had asked similar questions, but I
didn't see any answers.

As my program iterates along, it will write all the 0d, 1d, and 2d data 
at each step, but only writes the large 3 dimensional arrays at selected
steps (to conserve disk space).  There seems to be a built in constraint
at the core of the library that the amount of data for each step in the
unlimited dimension will be the same (which means low overhead and fast
calculation of the location of data).  The best you can do is to not
write or initialize a variable for a particular step, which leaves a
hole in the file.  Almost any sort of file copy will then fill out this
hole with nulls.  This could make the difference between a 5 MByte
output file and a 100 MByte output file (and we do a lot of runs).

I've looked at HDF and found that it does support this sort of use (it
seems to use linked lists for the unlimited dimensions).  I've also
found it has poor documentation, adds a gratuitous 15-30% disk space
penalty, has primitive and ugly utilities, and has limited post
processors.  The best utility to come with HDF is ncdump (which has only
limited understanding of HDF files).  The only other good point is the
web browser, but that can be used on netCDF files anyway.  Unfortunately, 
if we settle on HDF now, we probably won't be able to switch back to 
netCDF at a later date.

Am I missing something?  Is there some way to do what I want with the
current netCDF?  Is the rumored multi-unlimited dimension version of 
netCDF close to ready?  beta?  something being considered for next year? 
Can the HDF tools really be as bad as they look?  (it feels like I've
stepped back ten years whenever I use them)




Peter Halvorson
Siemens Power Corp
pjh@xxxxxxxxx