NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
On 11/10/16 4:01 PM, Liam Forbes wrote:
Hello! We are installing netCDF 4.4.1 w/ HDF5 1.8.17 on our new Intel based cluster. We've noticed the read performance on this cluster using ncks is extremely slow compared to a couple other systems. For example, parsing a file on our lustre 2.1 based filesystem takes less 8 seconds on our Cray XK6-200m. Parsing the same file on the same filesystem on our new cluster is taking 30+ seconds, with most of that time apparently spent reading in the file. Cray (hostname fish): fish1:lforbes$ time ncks test.nc <http://test.nc> out.nc <http://out.nc> real 0m4.804s user 0m3.180s sys 0m1.300s Cluster (hostname chinook): n0:loforbes$ time ncks mod.nc <http://mod.nc> out.nc <http://out.nc> real 0m32.435s user 0m29.240s sys 0m1.936s
A question: are you using the same version of NCO in both cases? A while back (year or two) I encountered a performance issue that was due to NCO and not netCDF (I tried many a combination). Charlie fixed whatever was causing the slowdown a while back, so if you just built NCO it's probably not that, but I wanted to make sure.
Matt -- Matt Thompson, SSAI, Sr Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson
netcdfgroup archives: