[netcdfgroup] nccopy to compressed NetCDF-4 magnitudes slower on NetCDF 4.6.2+

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: [netcdfgroup] nccopy to compressed NetCDF-4 magnitudes slower on NetCDF 4.6.2+
• From: Mike Chen <mike.scchen@xxxxxxxxx>
• Date: Thu, 11 Jul 2019 14:52:00 +0800

Hi,
We have a storage shortage and we'd like to leverage the compression to
save space.
The idea works fine for some time.
However, I was testing different NetCDF versions and happens to find that
the nccopy to compressed NetCDF-4 file is magnitudes slower on version
4.6.0+.

The test method follows:
OS: CentOS 6.10
HDF5: 1.10.5
NetCDF: 4.4.1.1, 4.5.0, 4.6.2, 4.7.0, HPC-NetCDF 4.6.2.4 from NOAA
All building uses system-provided GCC suite 4.4.1.

The test is simple: nccopy a ~400MB classic NetCDF file to NetCDF-4 with:
nccopy -4 -d 1 -s input.nc output.nc4
With version v4.5.0 and lower, the process takes ~7 seconds.
HPC-NetCDF also gives similar speed.
(gzip-ing the same file takes similar time.)
But with version v4.6.2+, the copy can take ~25 minutes!

Our production system still uses 4.4.0, so the speed should be okay.
But I'd like to know more so I can plan the updates accordingly.

Mike Chen
Research Assistant / System Engineer
Dept. of Atmospheric Science
National Taiwan University

• Follow-Ups:
  • Re: [netcdfgroup] nccopy to compressed NetCDF-4 magnitudes slower on NetCDF 4.6.2+
    • From: Ed Hartnett