Re: [netcdfgroup] [EXTERNAL] ZFP compression for NetCDF4

• To: Leigh Orf <leigh.orf@xxxxxxxxx>, "netcdfgroup@xxxxxxxxxxxxxxxx" <netcdfgroup@xxxxxxxxxxxxxxxx>
• Subject: Re: [netcdfgroup] [EXTERNAL] ZFP compression for NetCDF4
• From: "Lynnes, Christopher S. (GSFC-5860)" <christopher.s.lynnes@xxxxxxxx>
• Date: Tue, 23 Mar 2021 16:59:01 +0000
• Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nasa.gov; dmarc=pass action=none header.from=nasa.gov; dkim=pass header.d=nasa.gov; arc=none
• Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tkLXHO29qvX533WaSRZaAc4KXCKJFo6Wq5dl8ejiXVI=; b=dG/2+UnSx3ZaWq2v2IdI9HWN3dSfgotvqZjENktK0p98B7PV6bgZ6Bq5RnwoTYNATBBrIrjyM6CeHEa8oA/vgSLJaVxf58XPF6nfb54VsTCTkYqJ1ltzxkECNVGjPEXZzhDwZEsob+C9qgw7YSMo+E55LqQ1K6cdTsjK0TYG+xYJzMp7OfiQ5MINJcwyijMsHiOAKuFR3K0EDtExpoN8k8vltV/hDPG+o0pY8TjpyyvbE7xFXHdPGPp6NKK6+1CCiA3ESA1vXMBcNKmjsDN1K/PCKSb0Y/H9FMjTPL0XzrMWoJaWXTNP7CbTZxVKpBJ/ksafDFTmssMIPZAVl6niCw==
• Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NxD4a+bLkUTIKGatI1a52uL3nd8KMG3qfm82nYrLEHNCRyJjWGM19SQM9yS+hWKEq3cBcgLN9kVkCuQSLhV5y9TSp231hNdrU5s2CH/7NVX/IRoNH8oC2WFkjCAYQXLQ29SFO0/VIy85KT2R2ODkxnb0CTqXSyN+ljwYUzY7EL3tNrZ3A9ZYDltTSUahKt2EKV6V03ArVL7YhYsNo1plHVG8JXK3PC1vovwA+URZZIj3AS+9n96C8EqIn9J2jtEv7uYDWiWgPNl60mAn4JhfU2pR2yd1J5RUlXeC/RUZc/EJa7kN6MqGi7yU5ey7fZKiznMc9MmQ1LJRHqzo6XwrUA==
• Authentication-results: gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=none action=none header.from=nasa.gov;

A variant on #3 might be to create your data in uncompressed netCDF4 and use 
h5repack command to compress the data with the user-defined filter option.

—
Christopher Lynnes   NASA/GSFC    mobile: 410-231-4573
“Perfection is achieved, not when there is nothing more to add, but when there 
is nothing left to take away.” A. de St. Exupery

On 3/23/21, 12:53, "netcdfgroup on behalf of Leigh Orf" 
<netcdfgroup-bounces@xxxxxxxxxxxxxxxx<mailto:netcdfgroup-bounces@xxxxxxxxxxxxxxxx>
 on behalf of leigh.orf@xxxxxxxxx<mailto:leigh.orf@xxxxxxxxx>> wrote:

Hello,

I'm a long time user of HDF and netcdf. I create lots of data in my research 
(severe weather simulation) and ZFP compression in HDF5 files has been great. 
I'd like to be able to create netcdf4 files (preferably using the netcdf API) 
with 3D fields compressed using the lossy ZFP compression algorithm. I know 
that netcdf4 is HDF5 so it's certainly possible.

I'm wondering whether anyone could provide any pointers. I see 3 options really.

1. Create ZFP compressed netcdf4 files using the normal netCDF-C API (my code 
is mostly C)
2. Create ZFP compressed netcdf4 files using the HDF5 API (including all the 
HDF5 special sauce that makes netcdf4 netcdf4)
3. Use a version of nccopy that understands ZFP

Are any of these options possible right now? If not where are we on efforts to 
include ZFP into netCDF? I did grab the latest code on github and saw 
references to ZFP but no filter code.

Leigh Orf
UW-Madison

• References:
  • [netcdfgroup] ZFP compression for NetCDF4
    • From: Leigh Orf