Re: [netcdfgroup] What triggers a whole-file copy operation when modifying attributes?

• To: "Bentley, Philip" <philip.bentley@xxxxxxxxxxxxxxxx>
• Subject: Re: [netcdfgroup] What triggers a whole-file copy operation when modifying attributes?
• From: "Rob Latham" <robl@xxxxxxxxxxx>
• Date: Wed, 11 Jul 2012 09:00:00 -0500

On Wed, Jul 11, 2012 at 10:45:30AM +0100, Bentley, Philip wrote:
> My current understanding is that if one increases the length of a netCDF
> attribute then this will trigger a costly rewrite operation on the whole
> file since the header section is of fixed size. At least for netCDF-3
> files, I think.
> 
> But does this file rewrite penalty apply if the *net* change in header
> length arising from a series of attribute changes is zero or less?  Put
> another way, is the basic constraint the overall length of the header
> section, or the length of the individual attribute that one is planning
> to change ?
> 
> Some simple tests using ncatted suggest that it's the former - the
> length of the header section - though it would be useful to have this
> confirmed or refuted.

Your tests are accurate.  when exiting define mode, if the header is
"too big" it will trigger a rewrite.

> For example, I make changes to a series of text attributes such that
> they are now all shorter in length. This presumably frees up space in
> the header section. I then increase the length of some other attribute
> (perhaps a vector of ints). But that increase in length is accommodated
> by the space freed up earlier. So in this case a file rewrite/copy
> operation is not triggered, right?
> 
> Lastly, I notice that it's possible to add some padding to the header
> section using the h_minfree parameter to the nc__enddef() function.  Is
> it possible therefore to query the current amount of free space in the
> header section? I couldn't immediately see a function for doing that.

I don't know of one either.  You can poke around the internal data
structures if you are comfortable piercing the abstraction layer, but
that's probably not a great idea :>

==rob

-- 
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA

• References:
  • [netcdfgroup] What triggers a whole-file copy operation when modifying attributes?
    • From: Bentley, Philip