Re: [netcdfgroup] simultaneous NetCDF writes to same file

• To: Kristopher Bedka <kristopher.m.bedka@xxxxxxxx>
• Subject: Re: [netcdfgroup] simultaneous NetCDF writes to same file
• From: Russ Rew <russ@xxxxxxxxxxxxxxxx>
• Date: Tue, 29 May 2012 14:29:12 -0600

Hi Kristopher,

> I am processing a large volume of satellite data where multiple
> processes could be simultaneously writing data to the same netcdf
> file.   This has not been supported in previous NetCDF versions and
> I've gotten fatal errors when two simultaneous writes conflicted.  I
> now understand that recent NetCDF versions do support this
> functionality.  Could someone tell me or provide an example of what I
> need to do (i.e. new
> function calls, options in netcdf open, etc...) to make this work for
> me?    I've tried the pnetcdf package does not support chunking which
> I need to internally compress these files.

No, sorry, it's not supported in current netCDF versions either.
NetCDF-4 uses HDF5 as its storage layer, and HDF5 does not support
compression with parallel access, as explained here:

  http://www.hdfgroup.org/hdf5-quest.html#p5comp

--Russ

• Follow-Ups:
  • Re: [netcdfgroup] simultaneous NetCDF writes to same file
    • From: Rob Latham

• References:
  • [netcdfgroup] simultaneous NetCDF writes to same file
    • From: Kristopher Bedka