Re: [netcdfgroup] chunking and concatenating in NCO version 4.0.3

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: Re: [netcdfgroup] chunking and concatenating in NCO version 4.0.3
• From: Charlie Zender <zender@xxxxxxx>
• Date: Tue, 07 Sep 2010 21:14:04 -0700

Jennifer,

I am cross-posting this response to the netcdf-group,
since you raise some issues that might interest a wider audience.
Please post followups to the sourceforge site only.

First, NCO 4.0.1 introduced a simpler method to turn fixed dimensions
into record dimensions:

ncks --mk_rec_dmn time in.nc out.nc

Details are here: http://nco.sf.net/nco.html#mk_rec_dmn
This saves a few lines and significantly reduces disk access.
The preferred way to attack your problem is to change time into the
record dimension using this command once per input file.

The second issue is what we call "re-basing" the time coordinate:

http://nco.sf.net/nco.html#time_rebase

ncrcat should do this automatically once your input files have time
as the record dimension and UDUnits-legible time-since attributes.
If ncrcat does not, then that is a bug to examine separately.

The final issue is chunking. Your case does not fit a pre-defined
chunking map so you must manually specify your chunksizes. Denis shows
the correct syntax:

--cnk_plc g2d --cnk_dmn lev,1 --cnk_dmn lat,361 --chk_dmn l,720
--chk_dmn time,1

Note that your case seems to fit chunking map "rd1" except for lev,
which you wish to be 1 not 3. So I think this will work:

--cnk_map rd1 --cnk_dmn lev,1

i.e., use rd1 for everthing except lev. Your original command used
--cnk_scl which accepts _only one argument_ (you gave five, separated
by commas). cnk_scl means "use this same identical chunksize for every
dimension". The fact that your syntax was rewarded with a core dump
rather than a helpful syntax error is a bug which we will fix (TODO
nco

Only ncks and ncecat obey chunking arguments in version 4.0.3.
ncrcat, and all the operators except ncwa, also have this capability
in the current main trunk code, though it is "beta quality", because I
only added it Monday. All operators one would expect to will support
the chunking arguments in NCO 4.0.4.

As to your question

> Can another policy be implemented that tries to maintain current
> chunksize?

That seems like a reasonable suggestion.
There may be a more general need for a chunk map that means
"set the chunksizes of the first K dimensions equal to 1, and the
chunksizes of the remaining dimensions equal to the dimension
size". We'll look into supporting both of these.

Finally, I have nothing to offer about this behavior

"Changing the time axis to a record dimension more than doubles the
file size!"

other than it may not occur with netCDF3 (try it and see!).
netCDF4 has more internal structure than netCDF3.
NCO has no real control over filesizes---we just write data through
the API and netCDF internals determine the filesize.

Good luck,
Charlie
-- 
Charlie Zender, Department of Earth System Science
University of California, Irvine (949) 891-2429 :)