[netcdfgroup] Use the new quantize feature to get lossy compression of floating point data...

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: [netcdfgroup] Use the new quantize feature to get lossy compression of floating point data...
• From: Edward Hartnett - NOAA Affiliate <edward.hartnett@xxxxxxxx>
• Date: Thu, 13 Oct 2022 13:11:57 +0200
• Arc-authentication-results: i=1; mx.google.com; dkim=pass header.i=@noaa.gov header.s=google header.b=JJpQYtep; spf=pass (google.com: domain of edward.hartnett@xxxxxxxx designates 209.85.220.41 as permitted sender) smtp.mailfrom=edward.hartnett@xxxxxxxx; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=noaa.gov
• Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=to:subject:message-id:date:from:mime-version:dkim-signature; bh=2kDYElyLPmWrfDb/b19p2sCE8v8DtDiFF0TB7KezA8c=; b=PHGLehwI377YugpCzvT547flcUH4LAeUHDXdht2tMhUZZ4Ip3kA8ZRCH50zc99njGc u+kj4Wjz9yREuCwXpnFWM1jxk4VYZzLz8xmBoJhOJcRMvQj/wbzRqcikKlqGXQblQAFK 3g4kQC+V0mkffbwV6/yOQeboQoRCc/HXnRTzUhehvK0PLK/xtceZ9Ui3YAPaAgPxwAZ2 IkaicRQTJ4tZNHItw0VtyIHwW1OQ1IV9VylreoJFKRsV1XytDygq1yqFirVV8jcdjcML jEwieUCJuy5hW4HShqFr7ur2hIinvcG560E41HZXBPkqBmNzdAoCX81qWxuzVCTTvIk6 21Ww==
• Arc-seal: i=1; a=rsa-sha256; t=1665659530; cv=none; d=google.com; s=arc-20160816; b=kj05JsTd6QJeLdiZfOjBhNLW+SbrRNNCychYqHepKXwFj/HiWz68AODg0h+eOcuOjl UlymTYwhMfoC21if6QyroOSTFArskLHKXi/2l2hEv/BDqUUssXTtiKg4LeVyAnDzWrZH c2yWPAcE8UmIr2/IlNINiGstAwo5/M+I3RkgGqFgHaDJb3U7E0YZXjUZndeAUdeqhJTC uWKbz49xNjtKONbU9udlebh0xheU/v1EuLVrYggIgaFWNdkQAY5RF3jMpvPlYwwtJzxI 20fU7ETDxbgPMLRb2sOfWjMtL8x+od5iGINn+veaoS/Y/MHYXmBbbDhQCzUj9VXajt6X xoNQ==
• Authentication-results: vscanx3.ucar.edu; dkim=pass (2048-bit key, unprotected) header.d=noaa.gov header.i=@noaa.gov header.a=rsa-sha256 header.s=google header.b=JJpQYtep
• Authentication-results: mx.google.com; dkim=pass header.i=@noaa.gov header.s=google header.b=JJpQYtep; spf=pass (google.com: domain of edward.hartnett@xxxxxxxx designates 209.85.220.41 as permitted sender) smtp.mailfrom=edward.hartnett@xxxxxxxx; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=noaa.gov

Howdy Fellow NetCDFers!

The most recent release of the netcdf-c and netcdf-fortran libraries
include support for the quantize feature - this allows you to set the
number of significant digits for floating point variables. Bits not
required to maintain the specified precision are rounded, and this greatly
improves subsequent compression.

To get lossy compression, use the quantize feature with your floating point
variables, and then turn on either zlib or zstandard compression. The
result will be lossy, because your data will change. But the resulting file
will be far smaller, resulting in lower storage costs and transmission
times, and allowing both writing and reading programs to work more quickly.

Here's a presentation Charlie Zender and I did for the last EGU:
https://www.researchgate.net/publication/360815823_EGU22-13259_Adding_Quantization_to_the_NetCDF_C_and_Fortran_Libraries_to_Enable_Lossy_Compression_1

Here's the on-line documentation for the quantize feature in netCDF:
https://docs.unidata.ucar.edu/netcdf-c/current/md_quantize.html

I'm happy to answer any questions or help explain these new features to
Earth and space science data producers.

Keep on NetCDFing!
Ed Hartnett