Re: [netcdfgroup] nccopy to compressed NetCDF-4 magnitudes slower on NetCDF 4.6.2+

• To: Mike Chen <mike.scchen@xxxxxxxxx>
• Subject: Re: [netcdfgroup] nccopy to compressed NetCDF-4 magnitudes slower on NetCDF 4.6.2+
• From: Ed Hartnett <edwardjameshartnett@xxxxxxxxx>
• Date: Thu, 11 Jul 2019 09:01:28 -0600

Thanks for finding this and letting us know!

I have elevated this to an issue on GitHub so it gets proper attention:
https://github.com/Unidata/netcdf-c/issues/1436

Further discussion would be most helpful if it were in the issue, where we
can easily track it.

Keep on netCDFing!

Ed Hartnett

On Thu, Jul 11, 2019 at 7:51 AM Mike Chen <mike.scchen@xxxxxxxxx> wrote:

> Hi,
> We have a storage shortage and we'd like to leverage the compression to
> save space.
> The idea works fine for some time.
> However, I was testing different NetCDF versions and happens to find that
> the nccopy to compressed NetCDF-4 file is magnitudes slower on version
> 4.6.0+.
>
> The test method follows:
> OS: CentOS 6.10
> HDF5: 1.10.5
> NetCDF: 4.4.1.1, 4.5.0, 4.6.2, 4.7.0, HPC-NetCDF 4.6.2.4 from NOAA
> All building uses system-provided GCC suite 4.4.1.
>
> The test is simple: nccopy a ~400MB classic NetCDF file to NetCDF-4 with:
> nccopy -4 -d 1 -s input.nc output.nc4
> With version v4.5.0 and lower, the process takes ~7 seconds.
> HPC-NetCDF also gives similar speed.
> (gzip-ing the same file takes similar time.)
> But with version v4.6.2+, the copy can take ~25 minutes!
>
> Our production system still uses 4.4.0, so the speed should be okay.
> But I'd like to know more so I can plan the updates accordingly.
>
> Mike Chen
> Research Assistant / System Engineer
> Dept. of Atmospheric Science
> National Taiwan University
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web.  Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit:
> http://www.unidata.ucar.edu/mailing_lists/

• References:
  • [netcdfgroup] nccopy to compressed NetCDF-4 magnitudes slower on NetCDF 4.6.2+
    • From: Mike Chen