[netcdfgroup] better and faster compression are available in the new releases of netCDF C and Fortran libraries

• To: NetCDF Group List <netcdfgroup@xxxxxxxxxxxxxxxx>
• Subject: [netcdfgroup] better and faster compression are available in the new releases of netCDF C and Fortran libraries
• From: Ed Hartnett <edwardjameshartnett@xxxxxxxxx>
• Date: Tue, 9 Aug 2022 12:12:10 -0600

All:

The most recent releases of netcdf-c and netcdf-fortran support two new
compression features which will be very helpful at exascale:

* quantization - specify a number of significant digits for an NC_FLOAT or
NC_DOUBLE variable - excess bits are zeroed out, make compression much more
effective.
* zstandard compression - similar to zlib, but faster and better, zstandard
can now be used instead of deflate.

For more info on quantization see EGU extended abstract: "Adding
Quantization to the NetCDF C and Fortran Libraries to Enable Lossy
Compression
<https://www.researchgate.net/publication/360814804_Adding_Quantization_to_the_NetCDF_C_and_Fortran_Libraries_to_Enable_Lossy_Compression?_sg%5B0%5D=Q4tNkRMKR7_Ycz28KNTlzVgcYzZt7Ija-gMHZyC8597D2_Ibl4viGUwn4pv7Q1AKe7TSgSh0WoYzyHQK2qxp8Mr_ZjQjwRCBINR3RgPu.iOYXl273SL6mA_jxRaWR0wh6cl-c5UDLlqa3Ja8SdKaR6ngsfcPzUII0hQ2VHNHFu_4cmfXdN1ycpRVXd2Ay6g>"


To use zstandard compression, call function nc_def_var_zstandard()
<https://docs.unidata.ucar.edu/netcdf-c/current/netcdf__filter_8h.html#a01c702426610d3cd335da1430dfc9326>
 .

To use quantization, call function nc_def_var_quantize()
<https://docs.unidata.ucar.edu/netcdf-c/current/group__variables.html#ga669e4b23b9b7fb321b995982972cdd93>

Let me know if you need help.

Keep on NetCDFing!

Ed Hartnett