Re: Compressing netcdf files

• To: blincoln <blincoln@xxxxxxxxxx>
• Subject: Re: Compressing netcdf files
• From: John Caron <caron@xxxxxxxxxxxxxxxx>
• Date: Tue, 08 Aug 2000 07:14:37 -0600

blincoln wrote:
> 
> On Mon, 07 Aug 2000 16:20:36 -0600, John Caron wrote:
> 
> >this is very impressive; i wonder if this file happens to have a lot of
> >zeroes or missing data or something?
> >
> >floating point is known to be difficult to compress due to the fact the
> >low order bits are essentially random.
> 
> I haven't worked with bzip2 on many types of data yet, but I have
> been thoroughly impressed with it in several areas, compressing
> dramatically better than zip, compress, or gzip in about the same time
> as gzip (somewhat slower than zip) and decompressing pretty quickly.
> 
> Although when I tried it on a couple of several MB netCDF files I had
> (arps runs) I didn't see any real improvement over gzip or zip,
> I'd say its definitely worth trying if you have any in-house
> compression needs.. it isn't heavily used yet, but I have seen lots
> of systems start to use it in the last couple months because of the really
> substantial improvement in compression for some applications.
> 
> bcl

The compression ratios must be data dependent, but bzip2 is clearly
better than others on common netcdf files.  Russ sent this result for
some of our ldm files:

 size  compress  gzip  bzip2
 31Mb   2.54     4.13  6.04
  4Mb   3.63     4.47  7.82
 16Mb   3.58     4.32  7.36

• References:
  • Re: Compressing netcdf files
    • From: blincoln