NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
I definitely appreciate the suggestions and help that folks have provided thus far. I think I've made some progress narrowing down what the issue might be. This week I've trying different chunking policies and maps with NCO, and I also starting trying converting file formats. I think I've found that the slow performance only occurs when using the newer NetCDF4+HDF5 formats. Using the "classic" and "64-bit-offset" formats, my test ncks operation's performance on our new cluster is _much_ closer to the other systems. Interestingly, using the "classic" format in my test ncks operation on our Cray resulted in cutting the run time almost in half. Here's an example from the new cluster. loforbes$ ls -l test.nc -rw------- 1 loforbes staff 661565784 Nov 16 13:48 test.nc Unconverted: loforbes$ time ncks test.nc out.nc real 0m35.895s user 0m29.272s sys 0m1.815s Converted to "classic": loforbes$ time nccopy -k classic test.nc chinook_classic.nc real 0m6.185s user 0m2.170s sys 0m3.875s loforbes$ time ncks chinook_classic.nc out_classic.nc real 0m4.724s user 0m0.658s sys 0m3.908s loforbes$ ls -l *.nc [...cut...] -rw------- 1 loforbes staff 661302172 Nov 18 10:38 chinook_classic.nc [...cut...] -rw------- 1 loforbes staff 661302240 Nov 18 10:41 out_classic.nc [...cut...] I also noticed that no matter which format I converted the file to, the time to run nccopy was pretty similar, between 6 and 7 seconds. So these two things make me start to think the issue is at the HDF5 level. Could that be right? Like NetCDF when I started exploring this, I don't know much about HDF5. Is this the right place to ask for further suggestions? Should I continue trying to modify the chunking, but using HDF5 utilities instead of NCO utilities? I've already asked the researchers if there's anything about their data that requires using NetCDF4+HDF5 as the format. From reading the NetCDF documentation, I assume the fact that I was able to convert the file when making the copy means there isn't something specific to the newer format(s) in the data. Have a good weekend. -- Regards, -liam -There are uncountably more irrational fears than rational ones. -P. Dolan Liam Forbes loforbes@xxxxxxxxxx ph: 907-450-8618 fax: 907-450-8601 UAF Research Computing Systems Senior HPC Engineer LPIC1, CISSP
netcdfgroup archives: