NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
On 10/27/2014 11:30 PM, Chris Barker - NOAA Federal wrote:
On Oct 27, 2014, at 8:46 PM,However, you don't want chunks to be very small or very large just for the sake of efficiency. The ideal size and shape for a chunk is dictated by what access patterns you anticipate to be common, to make each access one chunk.Indeed, but I suspect that you'd hardly notice the difference between large and really large chunk sizes. For instance a chunk larger that your disk cache would probably buy you nothing. What's a typical disk cache these days? I have no idea. But I'll guess you won't see a noticeable drop in performance for a multi-gigabyte read if you use a 10MB or so chunk size.
close. Consider Mira, our large Blue Gene with GPFS. 16 MiB is a good size for "go as fast as you can" -- on that file system the block size is 8 MiB.
Some (small) multiple of the underlying file system block size is probably as far as you need to go. Certainly less than 100 MiB in 2014 (though that number is likely to creep upwards over time)
==rob
If you do any testing, please post your results here. -Chris--Russ On Mon, Oct 27, 2014 at 12:13 AM, Samrat Rao <samrat.rao@xxxxxxxxx <mailto:samrat.rao@xxxxxxxxx>> wrote: Hi, I am using netCDF-fortran-4.1.1 with netCDF-4.3.2. I get an error 'Bad chunk sizes' when the 3D data size is more than a certain number. I am unable to understand the cause of this error. My OS is CentOS 6.6. The errors are the same even if is use netCDF (v 4.1.1) supplied by the CentOS community. i am attaching a minimal example. Please let me know if there are any errors from my side. Thanks, Samrat. -- Samrat Rao Research Associate Engineering Mechanics Unit Jawaharlal Centre for Advanced Scientific Research Bangalore - 560064, India _______________________________________________ netcdfgroup mailing list netcdfgroup@xxxxxxxxxxxxxxxx <mailto:netcdfgroup@xxxxxxxxxxxxxxxx> For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/ -- Samrat Rao Research Associate Engineering Mechanics Unit Jawaharlal Centre for Advanced Scientific Research Bangalore - 560064, India _______________________________________________ netcdfgroup mailing list netcdfgroup@xxxxxxxxxxxxxxxx <mailto:netcdfgroup@xxxxxxxxxxxxxxxx> For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/_______________________________________________ netcdfgroup mailing list netcdfgroup@xxxxxxxxxxxxxxxx For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/
-- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA
netcdfgroup archives: