Re: [netcdfgroup] NetCDF-4 filesize question.

• To: Kent Yang <myang6@xxxxxxxxxxxx>
• Subject: Re: [netcdfgroup] NetCDF-4 filesize question.
• From: Chris Barker - NOAA Federal <chris.barker@xxxxxxxx>
• Date: Tue, 15 Apr 2014 07:43:09 -0700

On Apr 14, 2014, at 6:31 PM, Kent Yang <myang6@xxxxxxxxxxxx> wrote:

  I don’t think your programs cause this. For unlimited dimensions, the
HDF5 chunking storage is used. There are space overheads for chunks,
especially when the  chunk size is small.


Exactly -- very small chunk sizes can make for huge files. But this seems
like a lot of overhead to me.

Did you use the defaults for chunk sizes? I would have thought they'd do
pretty well for this case, but it's certainly worth trying setting them by
hand to see what you get.

Some googling will find you some good advice, but this is a good start:

http://www.unidata.ucar.edu/blogs/developer/en/entry/chunking_data_why_it_matters

Chris

One way to mitigate this issue  is to increase the chunk size and then to
use compression.



Kent



*From:* netcdfgroup-bounces@xxxxxxxxxxxxxxxx [mailto:
netcdfgroup-bounces@xxxxxxxxxxxxxxxx] *On Behalf Of *Wei Huang
*Sent:* Monday, April 14, 2014 4:38 PM
*To:* NetCDF Mailing List
*Subject:* [netcdfgroup] NetCDF-4 filesize question.



Hello group,



I am doing some NetCDF-4 test, and find when I use fixed, and unlimited
dimensions cause file size

changed dramatically.



Below is a list of files (with dimension 5 x 10 x 73 x 144, with few group
names):



1. has one unlimited dimension (the most left one)

2. has two unlimited dimensions (the left two)

3. has fixed dimensions.



-rw-r--r--   1 huangwei  CIT\Domain Users   6312828 Apr 14 15:25
NCLcreatedNC4.nc.1unlimited

-rw-r--r--   1 huangwei  CIT\Domain Users  10508612 Apr 14 15:24
NCLcreatedNC4.nc.2unlimited

-rw-r--r--   1 huangwei  CIT\Domain Users   2112758 Apr 14 15:26
NCLcreatedNC4.nc.fixed





My question is: does our program (has issues which) cause the file size
difference,

or, is it NetCDF4 which needs the extra space (for unlimited dimension)?



Thanks,





Wei Huang

_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit:
http://www.unidata.ucar.edu/mailing_lists/

• Follow-Ups:
  • Re: [netcdfgroup] NetCDF-4 filesize question.
    • From: Wei Huang

• References:
  • [netcdfgroup] NF90_CHAR vs NF90_STRING for attribute values
    • From: Mark Hadfield
  • Re: [netcdfgroup] NF90_CHAR vs NF90_STRING for attribute values
    • From: Mark Hadfield
  • [netcdfgroup] NetCDF-4 filesize question.
    • From: Wei Huang
  • Re: [netcdfgroup] NetCDF-4 filesize question.
    • From: Kent Yang