NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
On Thu, Jan 9, 2014 at 3:28 PM, Charlie Zender <zender@xxxxxxx> wrote: > I read with interest your discussion on chunking. > I added Chris's suggestion to NCO's supported chunking options. > > NCO 4.4.0 now implements six "chunking maps" > http://nco.sf.net/nco.html#cnk > nice! One small doc comment: """ Unchunking Definition: Unchunk all variables possible. The HDF5 storge layer requires that record variables (i.e., variables that contain at least one record dimension) must be chunked. Also variables that are compressed or use checksums must be chunked. """ Unlimited dimensions must be chunked as well. Not sure if NCO preserves those. And some thoughts: I may be mis-interpreting some of this (and not totally sure what a "record dimension" is), but """ Chunksize Equals Dimension Size except Record Dimension Definition: Chunksize equals dimension size except record dimension has size one. Explicitly specify chunksizes for particular dimensions with ‘--cnk_dmn’ option. cnk_map key values: ‘rd1’, ‘cnk_rd1’, ‘map_rd1’ Mnemonic: Record Dimension size 1 """ if you had a 1-d variable of records, would that mean chunks equal the record size? 'cause that would be way too small in the common case. """ Chunksize Lefter Product Matches Scalar Size Specified Definition: The product of the chunksizes for each variable (approximately) equals the size specified with the ‘--cnk_scl’ option. This is accomplished by using dimension sizes as chunksizes for the rightmost (most rapidly varying) dimensions, and then “flexing” the chunksize of the leftmost (least rapidly varying) dimensions such that the product of all chunksizes matches the specified size. All dimensions to the left of and including the first record dimension define the left-hand side. This map was first proposed by Chris Barker. cnk_map key values: ‘lfp’, ‘cnk_lfp’, ‘map_lfp’ Mnemonic: LeFter Product """ That sounds good -- and thanks for the credit! Not so clear from the amount of time I've spent reading that, but what would be the default chunking for a 1-d unlimited variable? or a 2-d, with one dimension very small (Nx3, for instance)? Those were the use cases where the default chunking in netcdf4 killed us. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@xxxxxxxx
netcdfgroup archives: