[netcdfgroup] AGU paper/poster on compression and netCDF

• To: NetCDF Group List <netcdfgroup@xxxxxxxxxxxxxxxx>
• Subject: [netcdfgroup] AGU paper/poster on compression and netCDF
• From: Ed Hartnett <edwardjameshartnett@xxxxxxxxx>
• Date: Thu, 16 Dec 2021 05:37:01 -0700

Howdy NetCDF Users!

Compression is one of the most popular and useful features of netCDF/HDF5
files, and compress also plays an important role in the new Zarr format.

Yet the zlib library is showing it's age - it's slow!

There is a new, faster, drop-in replacement, zlib-ng. Give it a try to
speed up your processing with no code changes:
https://github.com/zlib-ng/zlib-ng.

Zlib provides lossless compression, but lossy compression can be much more
effective, at the cost of some changes to the data. With the new
nc_def_var_quantize()/nc_inq_var_quantize() functions (which will be in the
next netCDF release), you can now specify a number of significant digits
for a variable. Then, turn on compression to get lossy compression, but to
preserve that number of significant digits.

More on zlib-ng, quantization, and other compression topics can be found in
this extended abstract presented at the AGU this week. Co-authos include
Charlie Zender (of bitgroom fame), Ward Fisher and Dennis Heimbigner
(Unidata netCDF programmers), and Hang Lei, Brian Curtis, and Kyle
Gerheiser of NOAA EMC, where we have great interest in compressing our
data. ;-)

Get the extended abstract here:
https://www.researchgate.net/publication/357001251_Quantization_and_Next-Generation_Zlib_Compression_for_Fully_Backward-Compatible_Faster_and_More_Effective_Data_Compression_in_NetCDF_Files

Get a slideshow of the results here:
https://www.researchgate.net/publication/357000984_Quantization_and_Next-Generation_Zlib_Compression_for_Fully_Backward-Compatible_Faster_and_More_Effective_Data_Compression_in_NetCDF_Files

Keep on NetCDFing!
Ed Hartnett