Re: [netcdfgroup] parallel netcdf4 fails on nc_enddef when deflate-filter is used

• To: Heiko Klein <Heiko.Klein@xxxxxx>
• Subject: Re: [netcdfgroup] parallel netcdf4 fails on nc_enddef when deflate-filter is used
• From: Russ Rew <russ@xxxxxxxxxxxxxxxx>
• Date: Tue, 6 Jan 2015 07:46:19 -0700

Heiko,

HDF5 supports reading compressed data in parallel, but cannot write
compressed data in parallel.  The netCDF library should return a more
specific error than "HDF5 error" in this case.  Here's an explanation
<http://www.hdfgroup.org/hdf5-quest.html#p5comp> for why HDF5 doesn't write
compressed data in parallel.

--Russ

On Tue, Jan 6, 2015 at 5:26 AM, Heiko Klein <Heiko.Klein@xxxxxx> wrote:

> Hi,
>
> we are usually using compressed netcdf4 files for our forecast meteorology
> files. This changes our bottleneck from IO to CPU, since compression
> (usually deflate level 3) takes some time.
>
> To speed this up, I intended to use the mpi-enabled parallel netcdf4/hdf5.
> I've been trying with netcdf 4.3.0 and netcdf 4.3.2 and hdf5-1.8.12 and
> hdf5-1.8.11 (I tried 1.8.13 and 1.8.14, too, but that didn't compile
> because of removed MPIPOSIX in hdf5).
>
> In all cases, I managed to get parallel netcdf working, but I never
> managed to create a deflated variable with parallel netcdf. As soon as I
> set something like
>
>    nc_def_var_deflate(ncid, varid, 1, 1, 3)
>
> I get a 'HDF5 error' when I call nc_enddef the next (first) time. Removing
> the deflate command, everything works again. Please find attached your
> tst_parallel4 test-file with the added nc_def_var_deflate command.
>
> Best regards,
>
> Heiko
>
> --
> Dr. Heiko Klein                              Tel. + 47 22 96 32 58
> Development Section / IT Department          Fax. + 47 22 69 63 55
> Norwegian Meteorological Institute           http://www.met.no
> P.O. Box 43 Blindern  0313 Oslo NORWAY
>
> _______________________________________________
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit:
> http://www.unidata.ucar.edu/mailing_lists/
>

• Follow-Ups:
  • Re: [netcdfgroup] parallel netcdf4 fails on nc_enddef when deflate-filter is used
    • From: Heiko Klein

• References:
  • [netcdfgroup] parallel netcdf4 fails on nc_enddef when deflate-filter is used
    • From: Heiko Klein