[netcdfgroup] How to uncompress with nccopy in netCDF-C 4.6.0

• To: "netcdfgroup@xxxxxxxxxxxxxxxx" <netcdfgroup@xxxxxxxxxxxxxxxx>
• Subject: [netcdfgroup] How to uncompress with nccopy in netCDF-C 4.6.0
• From: Joaquin Rodriguez Guerra <Joaquin.RodriguezGuerra@xxxxxxxxxxxx>
• Date: Sun, 25 Feb 2018 12:06:08 +0000

Hi there,


With netCDF-C now supporting HDF dynamic filters, the behaviour of nccopy has 
slightly changed. In the past nccopy could be used to uncompress files 
previously compressed with custom HDF dynamic filters. Nccopy used to read 
<infile>, relying on HDF to invoke the decompression using the filter, and then 
it would create a new netCDF file, <outfile>, uncompressed. In version 4.6.0, 
nccopy also reads <infile> relying on HDF as before, but then writes the new 
<outfile> applying the filter options defined in <infile>. I guess this is the 
new default behaviour when not passing the -F argument .

Is there any option that can be passed in the -F argument to tell nccopy not to 
apply any compression algorithm?

Thanks! and keep up the good work! Dynamic filter write support is a very hice 
feature!

Here an example, with a sample.nc file previously compressed with a custom HDF 
filter.

jrgu@linux-ozpe:~/FiltersTest/Test> ncdump -h -s sample.nc
netcdf sample {
dimensions:
number_of_columns = 30 ;
number_of_rows = 60 ;
variables:
short effective_radiance(number_of_rows, number_of_columns) ;
effective_radiance:_Storage = "chunked" ;
effective_radiance:_ChunkSizes = 60, 30 ;
effective_radiance:_Endianness = "little" ;
effective_radiance:_Filter = "56782,2,1,60,30,16,1,0,0,0,0,0,0,0" ; // filterID 
+ filter options (cd values)
ubyte pixel_quality(number_of_rows, number_of_columns) ;
pixel_quality:_Storage = "chunked" ;
pixel_quality:_ChunkSizes = 60, 30 ;
pixel_quality:_Filter = "56782,1,1,60,30,8,1,0,0,0,0,0,0,0" ; // filterID + 
filter options (cd values)

// global attributes:
:_SuperblockVersion = 2 ;
:_IsNetcdf4 = 1 ;
:_Format = "netCDF-4" ;
}

I have added some debugging messages to the custom HDF dynamic filter and 
obtained the following when using nccopy:

* Nccopy in NetCDF-c 4.4 and 4.5: the decoding functionality in the filter is 
invoked once for each variable when reading in_file:

jrgu@linux-ozpe:~/FiltersTest/Test> ../netcdf-c-4.5.0/ncdump/nccopy sample.nc 
sample_nccopy45.nc
Deconding
CD VALUES 2, 1, 60, 30, 16, 1, 0, 0, 0, 0, ...
Components: 1, Lines: 60, Samples: 30
DataBytes: 2
Decompression successfull
Decode End: 3600

Deconding
CD VALUES 1, 1, 60, 30, 8, 1, 0, 0, 0, 0, ...
Components: 1, Lines: 60, Samples: 30
DataBytes: 1
Decompression successfull
Decode End: 1800

* Nccopy in NetCDF-c 4.6: the decoding functionality in the filter is invoked  
once for each variable when reading in_file. The encoding functionality in the 
filter is invoked once for each variable when writing out_file

jrgu@linux-ozpe:~/FiltersTest/Test> nccopy sample.nc sample_nccopy46.nc
Deconding
CD VALUES 2, 1, 60, 30, 16, 1, 0, 0, 0, 0, ...
Components: 1, Lines: 60, Samples: 30
DataBytes: 2
Decompression successfull
Decode End: 3600

Deconding
CD VALUES 1, 1, 60, 30, 8, 1, 0, 0, 0, 0, ...
Components: 1, Lines: 60, Samples: 30
DataBytes: 1
Decompression successfull
Decode End: 1800

Encoding
CD VALUES 2, 1, 60, 30, 2, 1, 60, 30, 16, 1, ...
Components: 1, Lines: 60, Samples: 30
DataBytes: 2
Pixel: 1800
ERROR: Error in jpeglsCompress: Parameter values are not supported by CharLS.
Compression failed
ERROR: Error during the JPEG-LS compression of the dataset.
Encode End -> Result: 0

Encoding
CD VALUES 1, 1, 60, 30, 1, 1, 60, 30, 8, 1
Components: 1, Lines: 60, Samples: 30
DataBytes: 1
Pixel: 1800
ERROR: Error in jpeglsCompress: Parameter values are not supported by CharLS.
Compression failed
ERROR: Error during the JPEG-LS compression of the dataset.
Encode End -> Result: 0

Cheers,
Joaquín


Any email message from EUMETSAT is sent in good faith but shall neither be 
binding nor construed as constituting a commitment by EUMETSAT, except where 
provided for in a written agreement or contract or if explicitly stated in the 
email. Please note that any views or opinions presented in this email are 
solely those of the sender and do not necessarily represent those of EUMETSAT. 
This message and any attachments are intended for the sole use of the 
addressee(s) and may contain confidential and privileged information. Any 
unauthorised use, disclosure, dissemination or distribution (in whole or in 
part) of its contents is not permitted. If you received this message in error, 
please notify the sender and delete it from your system .

• Follow-Ups:
  • [netcdfgroup] Ambiguity in netCDF Documentation regarding "NetCDF Classic Format Limitations"
    • From: Daniel Neumann
  • Re: [netcdfgroup] How to uncompress with nccopy in netCDF-C 4.6.0
    • From: Dennis Heimbigner