Re: [netcdfgroup] read performance slow compared to netCDF on other systems

• To: netCDF Mail List <netcdfgroup@xxxxxxxxxxxxxxxx>
• Subject: Re: [netcdfgroup] read performance slow compared to netCDF on other systems
• From: Charlie Zender <zender@xxxxxxx>
• Date: Sun, 13 Nov 2016 11:35:44 -0800

Liam,

If (and only if) your files are netCDF4/HDF5 then chunking
differences might explain the observed behavior.
Does the system blocksize differ on the new and old systems?
To find out, use "stat <any file>".
If the blocksizes differ, then "ncks in.nc out.nc"
reads/writes different numbers of times.
To adjust this behavior, read how NCO uses the system blocksize
to determine the chunksize of datasets:
http://nco.sf.net/nco.html#blocksize
You can override the defaults with different chunking policies,
as described in the manual, and examine the chunksizes with
ncks --cdl --hdn -m in.nc.
Chunking is the only factor I can think of that might cause
such a significant slow-down.

Charlie
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(

• Follow-Ups:
  • Re: [netcdfgroup] read performance slow compared to netCDF on other systems
    • From: Liam Forbes