Re: Data compression

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: Re: Data compression
• From: caron@xxxxxxxxxxxxxxxxxxx (John Caron)
• Date: Thu, 17 Sep 1992 14:38:17 MDT

I think that lack of compression is the major failing of the netcdf format.  I 
have
been looking at the "WMO sanctioned" grib format lately and I am appreciating 
what
a major advance netcdf represents over those older forms.

I think that compression deserves more thought. It may be that by adding a few
reasonable restrictions, it could be supported in a straightforward manner.  If 
the major
problem is to support random write capability, then restrict that capability.  
It is
much more common to write once and read many times.  Typically if you write 
again to a file, it is a time series, and you are appending.  In any case it 
would "be alright" to
have a seperate call or even a seperate program that would compress after the 
file
was completely written.  

Obviously using a unix compress gets you the same thing.  However, you want 
netcdf to
manage the uncompress "on the fly" so that you dont have to worry about it. 
Here, I dont
see where the difficulty would be, unless you are worried about efficiency, 
e.g. having to 
decompress the entire file to get a a single cross section. I will say that if 
your 
compression is good enough, you can easily save more in I/O time than you spend 
in
decompression.

I suppose you could do it "outside" netcdf, and implement a daemon that grabs 
the ncopen 
call, check if the file exists, if not, checks if file.Z exists, and 
uncompresses.  
Give the daemon a fixed amount of disk space so that when it has used all of 
its space, 
and needs to uncompress another file, it deletes the oldest files in its 
subdirectory, etc.