NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
On Fri, Dec 6, 2019 at 6:45 AM Amr, Mahmoud <mahmoud.amr@xxxxxxxxxxxxxxxxx> wrote: > Maybe you can shed some light on how to set the chunk cache size and how > to pick the chunk size? We used information provided by this article which > suggests the chunk size should be small to get good average performance in > all dimensions: > > > > > https://www.unidata.ucar.edu/blogs/developer/en/entry/chunking_data_why_it_matters > Chunking is tricky -- if you want to support all access patterns, you want your chunks to be "square", not small. squareness means that accessing any axis is equivalent. But the optimum size of the chunks has to do with other things, like the size of the disk cache, block size, etc. "When a user or a program makes a request for data, the hard drive pulls that data and the data around it on the platter and stores it all in the buffer." (https://www.techjunkie.com/what-is-hard-drive-cache/): So even if your program asks for only one byte, the disk is reading a much larger "chunk" of data into cache anyway -- ideally your HDF chunk size matches the chuck that the disk is putting in cache. Which is hard, because you probably don't know the nature of the hardware that will be used to access your files :-( However, in limited experimentation, I found that performance wasn't very sensitive to chunk sizes within a "reasonable" range: - very small chunks absolutely kill performance (and make the files larger than needed) - very large chunks also kill performance for non-optimum access patterns. The only thing to do (that I know of) is experiment with your "typical" hardware and access patterns. Another thought: do you really need access to ALL axes equally? maybe yes, maybe no. NOTE: I'm not really up on hardware, but 64 x 64 x 64 is 256k -- that is pretty darn small by today's standards. I'd start with at least 640x640x640 HTH, -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@xxxxxxxx
netcdfgroup archives: