Re: [netcdfgroup] ncgen should remove special attributes like _Shuffle

• To: Heiko Klein <Heiko.Klein@xxxxxx>
• Subject: Re: [netcdfgroup] ncgen should remove special attributes like _Shuffle
• From: Dave Allured - NOAA Affiliate <dave.allured@xxxxxxxx>
• Date: Mon, 8 Apr 2013 13:59:20 -0600

This problem is already fixed for the next Netcdf release, according
to the tracker.

http://www.unidata.ucar.edu/support/help/MailArchives/netcdf/msg11533.html
https://bugtracking.unidata.ucar.edu/browse/NCF-213

--Dave

On Mon, Apr 8, 2013 at 1:16 PM, Heiko Klein <Heiko.Klein@xxxxxx> wrote:
> Hi,
>
> newer versions of netcdf (tested with 4.2.1 and 4.1.1) allow the generation
> of netcdf4 files with shuffling, chunking and compression by using the
> attributes:
>
> `_Format', `_Fletcher32,  `_ChunkSizes',  `_Endianness',_DeflateLevel',
> `_Shuffle', and `_Storage'.
>
> Unfortunately, these attributes are not only used for creating the file, but
> are also written as normal attributes to the file.
>
> A simple example:
>
> netcdf test {
> dimensions:
>     time = UNLIMITED ; // (2 currently)
> variables:
>     short time(time) ;
>         time:_Storage = "chunked" ;
>         time:_ChunkSizes = 1 ;
>         time:_Shuffle = "true" ;
>         time:_DeflateLevel = 3 ;
> // global attributes:
>         :_Format = "netCDF-4 classic model" ;
> data:
>  time = 12, 24 ;
> }
>
>
> looks after a 'ncgen -b', and 'ncdump -hs test.nc' like:
>
> netcdf test {
> dimensions:
>         time = UNLIMITED ; // (2 currently)
> variables:
>         short time(time) ;
>                 time:_ChunkSizes = 1 ;
>                 time:_DeflateLevel = 3 ;
>                 time:_Shuffle = "true" ;
>                 time:_Storage = "chunked" ;
>                 time:_ChunkSizes = 1 ;
>                 time:_DeflateLevel = 3 ;
>                 time:_Shuffle = "true" ;
>                 time:_Endianness = "little" ;
>
> // global attributes:
>                 :_Format = "netCDF-4 classic model" ;
> }
>
>
> When using some further processing, e.g. 'nccopy -k 3 -d 9 test.nc
> test9.nc', test9.nc will have ambiguos _DeflateLevel settings:
>
> $ ncdump -hs test9.nc
> netcdf test9 {
> dimensions:
>         time = UNLIMITED ; // (2 currently)
> variables:
>         short time(time) ;
>                 time:_ChunkSizes = 1 ;
>                 time:_DeflateLevel = 3 ;
>                 time:_Shuffle = "true" ;
>                 time:_Storage = "chunked" ;
>                 time:_ChunkSizes = 1 ;
>                 time:_DeflateLevel = 9 ;
>                 time:_Endianness = "little" ;
>
> // global attributes:
>                 :_Format = "netCDF-4" ;
> }
>
>
> I guess, all those _Attributes should never be written to the file by ncgen.
> (Tested with netcdf-4.2.1.1 on linux (ubuntu 12.04 64bit)).
>
> Regards,
>
> Heiko
>
> _______________________________________________
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit:
> http://www.unidata.ucar.edu/mailing_lists/

• References:
  • [netcdfgroup] ncgen should remove special attributes like _Shuffle
    • From: Heiko Klein