Re: [netcdfgroup] NetCDF4 MPI-IO and compression

• To: Ed Hartnett <ed@xxxxxxxxxxxxxxxx>
• Subject: Re: [netcdfgroup] NetCDF4 MPI-IO and compression
• From: Rob Ross <rross@xxxxxxxxxxx>
• Date: Wed, 13 Feb 2008 14:18:56 -0600

That makes sense. -- Rob

On Feb 13, 2008, at 2:03 PM, Ed Hartnett wrote:

Rob Ross <rross@xxxxxxxxxxx> writes:

In defense of the HDF team, this is difficult capability to enable in

practice. This is because prior to compression you don't know how much

space a given write will require, so parallelizing writes (especially
if they aren't performed collectively) is not straightforward and
requires a significant amount of coordination.

Rob

Also I was mistaken when I wrote that parallel I/O and the deflate
filter can't be used together. This is only true for writing the data,
not reading it.

So you can read compressed data in parallel.

Thanks,

Ed
--
Ed Hartnett  -- ed@xxxxxxxxxxxxxxxx

• References:
  • [netcdfgroup] NetCDF4 MPI-IO and compression
    • From: Thomas Green
  • Re: [netcdfgroup] NetCDF4 MPI-IO and compression
    • From: Ed Hartnett
  • Re: [netcdfgroup] NetCDF4 MPI-IO and compression
    • From: Rob Ross
  • Re: [netcdfgroup] NetCDF4 MPI-IO and compression
    • From: Ed Hartnett