Re: overcoming netcdf3 limits

• To: Ed Hartnett <ed@xxxxxxxxxxxxxxxx>
• Subject: Re: overcoming netcdf3 limits
• From: robl@xxxxxxxxxxx (Robert Latham)
• Date: Tue, 24 Apr 2007 16:43:35 -0500

On Tue, Apr 24, 2007 at 02:30:39PM -0600, Ed Hartnett wrote:
> However, another question: have you considered using netCDF-4? It does
> not have the limits of the 64-bit offset format, and support parallel
> I/O, as well as a number of other features (groups, compound data
> types) which might be helpful in organizing really large data sets.

Hi Ed

The CDF-1 and CDF-2 file formats appear to be quite robust in the face
of client failures.  Greg S at least has observed file corruption with
the HDF5 file format during parallel I/O if a client dies at a
particular time.   As I understand it, it's hard to devise a solution
for the HDF5 file format that is both rock-solid robust *and* delivers
high-performance. 

In parallel-netcdf land we also like the CDF-1 and CDF-2 file formats:
they are easy to work with from an MPI-IO perspective.    Also, by
writing out all the metadata in define mode, there is very little
chance of file corruption for any failure in data mode. 

Thanks
==rob

-- 
Rob Latham
Mathematics and Computer Science Division    A215 0178 EA2D B059 8CDF
Argonne National Lab, IL USA                 B29D F333 664A 4280 315B

==============================================================================
To unsubscribe netcdfgroup, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================

• References:
  • overcoming netcdf3 limits
    • From: Robert Latham
  • Re: overcoming netcdf3 limits
    • From: Ed Hartnett