NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Dear Ted,
maybe I was shortening a bit too much:
netcdf test1_ {
dimensions:
pixel_x_0 = 11256 ;
pixel_y_0 = 256 ;
complex_0 = 2 ;
variables:
double data_complex_set_0(pixel_x_0, pixel_y_0, complex_0) ;
data_complex_set_0:title = "New Field" ;
data_complex_set_0:_Storage = "chunked" ;
data_complex_set_0:_ChunkSizes = 3752, 86, 1 ;
data_complex_set_0:_DeflateLevel = 9 ;
...
Further I tried different deflate levels. - all the same result.
Shuffle also makes no difference.
Best,
Frank
2013/12/18 Ted Mansell <ted.mansell@xxxxxxxx>
> What are the actual dimensions of the variable? The rightmost
> (fastest-varying) chunk dimension is 1, which might be a problem. If that
> is an unlimited dimension, then reordering the array would be a good idea,
> if possible. You might try turning the Shuffle filter on, but I don't know
> how much effect that would have. Are you setting the chunk dimensions
> yourself or relying on the default chunking scheme?
>
> On a side note, my experience is that deflatelevel=2 gives a good
> compromise between speed and compression. Higher values tend to yield only
> modestly better compression for the increased computation. Your mileage may
> vary!
>
> Cheers,
>
> -- Ted
>
> On Dec 18, 2013, at 11:37 AM, Frank Fuchs wrote:
>
> > I tried ncdump -s getting the follwing infos:
> >
> > variables:
> > ...
> > data_complex_set_0:_Storage = "chunked" ;
> > data_complex_set_0:_ChunkSizes = 3752, 86, 1 ;
> > data_complex_set_0:_DeflateLevel = 9 ;
> >
> > // global attributes:
> > ....
> > :_Format = "netCDF-4" ;
> >
> > Are those chunksizes meaningful?
> >
> > On a different thought. Does netcdf use zlib directly or via the HDF
> library?
> > Something could go wrong there as well, no?
> >
> > Thank you! Best,
> > Frank
> >
> >
> >
> >
> > 2013/12/18 Russ Rew <russ@xxxxxxxxxxxxxxxx>
> > Hi Frank,
> >
> > > Now I wanted to test compression using the cxx4 interface, enabling it
> by
> > > ncvar_data.setCompression(true,true,1) for the heaviest of my
> variables.
> > >
> > > However, even for a file filled with constants the files remain as big
> as before.
> > > Further tests using nccopy -d9 old.nca new.nca did not result in a
> modification of the file size.
> >
> > If you use an unlimited dimension, that may prevent compression,
> > because it means that each variable is divided into chunks for
> > compression, with one record per chunk. There is significant HDF5
> > space overhead for storing lots of tiny chunks, even if they can be
> > compressed.
> >
> > Two solutions include:
> >
> > 1. If you don't need the unlimited dimension any more, perhaps
> > because no more data will be appended to the files, then convert
> > the unlimited dimension into a fixed-size dimension, resulting in
> > all the values of each variable being stored contiguously, which
> > should be more compressible.
> >
> > 2. If you still need the unlimited dimension, then rechunk the data
> > before compressing it, so the compression can work on larger
> > chunks.
> >
> > The nccopy utility can be used for both of these approaches.
> >
> > For approach 1:
> >
> > $ nccopy -u orig.nc orig-u.nc # makes unlimited dimension
> fixed size
> > $ nccopy -d9 orig-u.nc orig-u-d9.nc # compresses result
> >
> > For approach 2, assuming you have a record dimension "t" with each chunk
> > a slice of only one t value:
> >
> > $ nccopy -c t/10 orig.nc orig-c.nc # chunks t dimension using 10
> instead of 1
> > $ nccopy -d9 orig-c.nc orig-c-d9.nc # compresses result
> >
> > --Russ
> >
> >
> > > --===============1981692180==
> > > Content-Type: multipart/alternative;
> boundary=047d7bdc99b29242bc04edc0db6b
> > >
> > > --047d7bdc99b29242bc04edc0db6b
> > > Content-Type: text/plain; charset=ISO-8859-1
> > >
> > > Hi,
> > >
> > > I managed to compile netcdf-4.3.0 using mingw-w64 gcc 4.8.1.
> > > All I had to disabale was DAP (I have no use for anyway).
> > >
> > > I tested that I can read and write netcdf files using the newly build
> .dll
> > > Now I wanted to test compression using the cxx4 interface, enabling it
> by
> > > ncvar_data.setCompression(true,true,1) for the heaviest of my
> variables.
> > >
> > > However, even for a file filled with constants the files remain as big
> as
> > > before.
> > > Further tests using nccopy -d9 old.nca new.nca did not result in a
> > > modification of the file size.
> > >
> > > Any advise?
> > >
> > > Best,
> > > Frank
> > >
> > > --047d7bdc99b29242bc04edc0db6b
> > > Content-Type: text/html; charset=ISO-8859-1
> > > Content-Transfer-Encoding: quoted-printable
> > >
> > > <div dir=3D"ltr">Hi,<div><br></div><div>I managed to compile
> netcdf-4.3.0 u=
> > > sing mingw-w64 gcc 4.8.1.</div><div>All I had to disabale was DAP (I
> have n=
> > > o use for anyway).</div><div><br></div><div>I tested that I can read
> and wr=
> > > ite netcdf files using the newly build .dll</div>
> > > <div>Now I wanted to test compression using the cxx4 interface,
> enabling it=
> > > by=A0</div><div>ncvar_data.setCompression(true,true,1) for the
> heaviest of=
> > > my variables.=A0<br></div><div><br></div><div>However, even for a
> file fil=
> > > led with constants the files remain as big as before.=A0</div>
> > > <div>Further tests using nccopy -d9 old.nca new.nca did not result in
> a mod=
> > > ification of the file size.</div><div><br></div><div>Any
> advise?</div><div>=
> > >
> <br></div><div>Best,</div><div>Frank</div><div><br></div><div><br></div>
> > > </div>
> > >
> > > --047d7bdc99b29242bc04edc0db6b--
> > >
> > >
> > > --===============1981692180==
> > > Content-Type: text/plain; charset="us-ascii"
> > > MIME-Version: 1.0
> > > Content-Transfer-Encoding: 7bit
> > > Content-Disposition: inline
> > >
> > > _______________________________________________
> > > netcdfgroup mailing list
> > > netcdfgroup@xxxxxxxxxxxxxxxx
> > > For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/m
> > > ailing_lists/
> > > --===============1981692180==--
> >
>
>
netcdfgroup archives: