Compressed netCDF and sparse data

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: Compressed netCDF and sparse data
• From: Jeremy Beal <jeremyb@xxxxxxxxxxxxx>
• Date: Tue, 30 Jun 1998 16:21:49 -0600

Greetings!

I recently noted with interest the April announcement
that a user had achieved transparent compression within
the netCDF library by patching the library and linking
with the zlib compression library. This might provide
a solution to a sparse data storage problem I am currently
looking at.

I have been unable to successfully access the source
of the netCDF 3.31 library patch via the link provided
at http://snow.cit.cornell.edu/noon/z_netcdf.html.
(The file download URL is
http://snow.cit.cornell.edu/noon/znetcdf.tar.gz)

I can successfully download the file with a shift-click,
however the resulting file cannot be uncompressed with
WinZip (undoes the gz OK but reports a corrupted .tar file),
nor can I get to the inner tar file by using gzip, which
reports "invalid compressed data -- CRC error". Am I getting
garbled by the Windows linefeed convention? I would expect
that everything is binary except the innermost files. We have
been unable to get to the file via anonymous FTP to related
sites. I have been unable to locate an email address for
Bill Noon.

Has anybody else been able to get to the patch code?
I am also unable to locate any source at the unidata
FTP site under the "contrib" subdirectory.

Has any consideration been given to adopting compressed
netCDF as a standard part of the netCDF package?

Another option we had considered for the sparse data
problem would involve using a netCDF file as a more generalized
"header" to store/recover access points within an XDR file, as a 
means of getting around the fixed record size limitation.

Thanks sincerely,

Jeremy Beal
jeremyb@xxxxxxxxxxxxx
Scientific Software-Intercomp, Inc.