[netcdfgroup] ZFP compression for NetCDF4

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: [netcdfgroup] ZFP compression for NetCDF4
• From: Leigh Orf <leigh.orf@xxxxxxxxxxxxx>
• Date: Tue, 23 Mar 2021 11:51:29 -0500

Hello,

I'm a long time user of HDF and netcdf. I create lots of data in my
research (severe weather simulation) and ZFP compression in HDF5 files has
been great. I'd like to be able to create netcdf4 files (preferably using
the netcdf API) with 3D fields compressed using the lossy ZFP compression
algorithm. I know that netcdf4 is HDF5 so it's certainly possible.

I'm wondering whether anyone could provide any pointers. I see 3 options
really.

1. Create ZFP compressed netcdf4 files using the normal netCDF-C API (my
code is mostly C)
2. Create ZFP compressed netcdf4 files using the HDF5 API (including all
the HDF5 special sauce that makes netcdf4 netcdf4)
3. Use a version of nccopy that understands ZFP

Are any of these options possible right now? If not where are we on efforts
to include ZFP into netCDF? I did grab the latest code on github and saw
references to ZFP but no filter code.

Leigh Orf
UW-Madison