NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Hi Russ, Thanks for the info. On reading your reply I too recollected reading somewhere that the maximum chunk size is 2Gb. I'll try to break down the chunk sizes --- i also did not know that the entire chunk is to be read into memory. --- Samrat. On Mon, Oct 27, 2014 at 11:53 PM, Russ Rew <russ@xxxxxxxxxxxxxxxx> wrote: > Hi Samrat, > > You're requesting chunks over 17GB in size. If you were to read even a > single value from a chunk of that size, it would require reading the full > chunk into memory first, since chunks are the atomic units of data access > for chunked datasets. > > The theoretical limit on the size of chunks is how much contiguous memory > can be allocated in one call, which is 2^31 on 32-bit systems and 2^63 on > 64-bit platforms. However, the practical limit is how much memory it's > possible to use and you're willing to use for an I/O buffer. I have a > recollection that the HDF5 library used as the storage layer for netCDF-4 > limits chunks to be smaller than 2 GB, but I can't find where that's > documented. Looking at the netCDF library code, I can't see an enforced > maximum size, other than the maximum value of the size_t type, but we > should add that to the documentation if it's not there. > > However, you don't want chunks to be very small or very large just for the > sake of efficiency. The ideal size and shape for a chunk is dictated by > what access patterns you anticipate to be common, to make each access one > chunk. > > --Russ > > On Mon, Oct 27, 2014 at 12:13 AM, Samrat Rao <samrat.rao@xxxxxxxxx> wrote: > >> >> Hi, >> >> I am using netCDF-fortran-4.1.1 with netCDF-4.3.2. I get an error 'Bad >> chunk sizes' when the 3D data size is more than a certain number. >> >> I am unable to understand the cause of this error. >> >> My OS is CentOS 6.6. The errors are the same even if is use netCDF (v >> 4.1.1) supplied by the CentOS community. >> >> i am attaching a minimal example. Please let me know if there are any >> errors from my side. >> >> Thanks, >> Samrat. >> >> -- >> >> Samrat Rao >> Research Associate >> Engineering Mechanics Unit >> Jawaharlal Centre for Advanced Scientific Research >> Bangalore - 560064, India >> >> _______________________________________________ >> netcdfgroup mailing list >> netcdfgroup@xxxxxxxxxxxxxxxx >> For list information or to unsubscribe, visit: >> http://www.unidata.ucar.edu/mailing_lists/ >> > > -- Samrat Rao Research Associate Engineering Mechanics Unit Jawaharlal Centre for Advanced Scientific Research Bangalore - 560064, India
netcdfgroup archives: