Re: [netcdfgroup] Compressed sizes of netCDF variables

• To: Timothy Patterson <Timothy.Patterson@xxxxxxxxxxxx>, "netcdfgroup@xxxxxxxxxxxxxxxx" <netcdfgroup@xxxxxxxxxxxxxxxx>
• Subject: Re: [netcdfgroup] Compressed sizes of netCDF variables
• From: Kent Yang <myang6@xxxxxxxxxxxx>
• Date: Mon, 19 May 2014 15:08:10 +0000
• Authentication-results: spf=none (sender IP is ) smtp.mailfrom=myang6@xxxxxxxxxxxx;

Not sure about the NetCDF-4 end but since your file is actually an HDF5 file, 
you can use h5dump -H -p to dump the header and the compression info. for each 
variable. 
Search STORAGE_LAYOUT for the output file. You can see something like:


DATASET "Image_14" {
         DATATYPE  H5T_STD_U8LE
         DATASPACE  SIMPLE { ( 700, 1000, 3 ) / ( 700, 1000, 3 ) }
         STORAGE_LAYOUT {
            CHUNKED ( 700, 1000, 3 )
            SIZE 72698 (28.887:1 COMPRESSION)
         }
         FILTERS {
            PREPROCESSING SHUFFLE
            COMPRESSION DEFLATE { LEVEL 1 }
         }

That SIZE info. tells you the compression ratio. The filters tell you the 
compression scheme you are using. Currently for NetCDF, only deflate and 
shuffle are available.

Kent

-----Original Message-----
From: netcdfgroup-bounces@xxxxxxxxxxxxxxxx 
[mailto:netcdfgroup-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Timothy Patterson
Sent: Monday, May 19, 2014 10:00 AM
To: netcdfgroup@xxxxxxxxxxxxxxxx
Subject: [netcdfgroup] Compressed sizes of netCDF variables

An example:

I have a netCDF file with 5 2-byte integer variables, each of dimensions 100 x 
100 but containing different values. The data file on disk has a size of 
(roughly) 100,000 bytes.

I then compress the variables individually using the in-built zip, and get file 
size of 50,000 bytes. So my average compression across all the variables is a 
factor of 2.

However, I'd like to monitor how well each individual variable has compressed. 
Is there a way to determine the compressed size of a variable within the netCDF 
file?

Thanks!

Tim Patterson

_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/ 

• References:
  • [netcdfgroup] Compressed sizes of netCDF variables
    • From: Timothy Patterson