[netcdfgroup] Use the new quantize feature to get lossy compression of floating point data...

To: netcdfgroup@xxxxxxxxxxxxxxxx
Subject: [netcdfgroup] Use the new quantize feature to get lossy compression of floating point data...
From: Edward Hartnett - NOAA Affiliate <edward.hartnett@xxxxxxxx>
Date: Thu, 13 Oct 2022 13:11:57 +0200
Authentication-results: vscanx3.ucar.edu; dkim=pass (2048-bit key, unprotected) header.d=noaa.gov header.i=@noaa.gov header.a=rsa-sha256 header.s=google header.b=JJpQYtep
Authentication-results: mx.google.com; dkim=pass header.i=@noaa.gov header.s=google header.b=JJpQYtep; spf=pass (google.com: domain of edward.hartnett@xxxxxxxx designates 209.85.220.41 as permitted sender) smtp.mailfrom=edward.hartnett@xxxxxxxx; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=noaa.gov

Howdy Fellow NetCDFers!

The most recent release of the netcdf-c and netcdf-fortran libraries
include support for the quantize feature - this allows you to set the
number of significant digits for floating point variables. Bits not
required to maintain the specified precision are rounded, and this greatly
improves subsequent compression.

To get lossy compression, use the quantize feature with your floating point
variables, and then turn on either zlib or zstandard compression. The
result will be lossy, because your data will change. But the resulting file
will be far smaller, resulting in lower storage costs and transmission
times, and allowing both writing and reading programs to work more quickly.

Here's a presentation Charlie Zender and I did for the last EGU:
https://www.researchgate.net/publication/360815823_EGU22-13259_Adding_Quantization_to_the_NetCDF_C_and_Fortran_Libraries_to_Enable_Lossy_Compression_1

Here's the on-line documentation for the quantize feature in netCDF:
https://docs.unidata.ucar.edu/netcdf-c/current/md_quantize.html

I'm happy to answer any questions or help explain these new features to
Earth and space science data producers.

Keep on NetCDFing!
Ed Hartnett