[netcdfgroup] Per variable chunking with nccopy

• To: "netcdfgroup@xxxxxxxxxxxxxxxx" <netcdfgroup@xxxxxxxxxxxxxxxx>
• Subject: [netcdfgroup] Per variable chunking with nccopy
• From: Aidan Heerdegen <aidan.heerdegen@xxxxxxxxxx>
• Date: Mon, 12 Jan 2015 06:01:43 +0000
• Authentication-results: spf=none (sender IP is ) smtp.mailfrom=aidan.heerdegen@xxxxxxxxxx;

Hi,

Before I throw myself into implementing this, has anyone else modified nccopy to
algorithmically determine chunking specs for variables, or allowed this to be
specified on a per variable basis?

I’m trying the approach documented by Russ Rew here:

http://www.unidata.ucar.edu/blogs/developer/en/entry/chunking_data_choosing_shapes

and have a python algorithm working for my 3 and 4 dimensional variables. I had 
hoped
to use nccopy and pass these chunking schemes to it, but as nccopy so 
succinctly describes 
in its own man page:

 "Note that nccopy requires variables that share a dimension to also share the 
chunk 
  size associated with that  dimension,  but the programming interface has no 
such 
  restriction.  If you need to customize chunking for variables independently, 
you will 
  need to use the library API in a custom utility program.”

and I have 3D and 4D variables in the same file which share dimensions.

I am thinking the easiest route may be for me to fork nccopy and implement this
optimal chunking algorithm internally. Does this sound like a good idea? I was 
hoping
someone else was already implementing something like this.

Thanks

Aidan