NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
On Tue, May 29, 2012 at 02:29:12PM -0600, Russ Rew wrote: > Hi Kristopher, > > > I am processing a large volume of satellite data where multiple > > processes could be simultaneously writing data to the same netcdf > > file. This has not been supported in previous NetCDF versions and > > I've gotten fatal errors when two simultaneous writes conflicted. I > > now understand that recent NetCDF versions do support this > > functionality. Could someone tell me or provide an example of what I > > need to do (i.e. new > > function calls, options in netcdf open, etc...) to make this work for > > me? I've tried the pnetcdf package does not support chunking which > > I need to internally compress these files. > > No, sorry, it's not supported in current netCDF versions either. > NetCDF-4 uses HDF5 as its storage layer, and HDF5 does not support > compression with parallel access, as explained here: Is there any chance you can compress at the application layer? Each processor takes it's local hunk of data, compresses it, then writes to the file. I admit, you will quickly find out why parallel writes with compression is not already implemented in these parallel I/O libraries! However, it's possible that at your application level, there may be ways to simplify the parallel, compressed writes problem that a general purpose library cannot use. ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA
netcdfgroup archives: