NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

Re: [netcdfgroup] [EXTERNAL] Re: NetCDF parallel I/O configurations

  • To: "Sjaardema, Gregory D" <gdsjaar@xxxxxxxxxx>, "Latham, Robert J." <robl@xxxxxxxxxxx>, "seanb@xxxxxxxx" <seanb@xxxxxxxx>
  • Subject: Re: [netcdfgroup] [EXTERNAL] Re: NetCDF parallel I/O configurations
  • From: Kent Yang <myang6@xxxxxxxxxxxx>
  • Date: Wed, 2 Mar 2016 22:01:32 +0000
  • Authentication-results: sandia.gov; dkim=none (message not signed) header.d=none; sandia.gov; dmarc=none action=none header.from=hdfgroup.org;
  • Spamdiagnosticmetadata: NSPM
  • Spamdiagnosticoutput: 1:23
I don't think pnetcdf from ANL uses the chunking technique as the HDF5 does. 
That may lead to bigger performance difference when some subset patterns get 
involved.

Kent

-----Original Message-----
From: netcdfgroup-bounces@xxxxxxxxxxxxxxxx 
[mailto:netcdfgroup-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Sjaardema, Gregory D
Sent: Wednesday, March 02, 2016 3:55 PM
To: Latham, Robert J.; seanb@xxxxxxxx
Cc: netcdfgroup@xxxxxxxxxxxxxxxx
Subject: Re: [netcdfgroup] [EXTERNAL] Re: NetCDF parallel I/O configurations


On 3/2/16, 2:49 PM, "netcdfgroup-bounces@xxxxxxxxxxxxxxxx on behalf of Latham, 
Robert J." <netcdfgroup-bounces@xxxxxxxxxxxxxxxx on behalf of robl@xxxxxxxxxxx> 
wrote:

>On Tue, 2016-02-23 at 17:13 +0000, Sean Byland wrote:
>> Hello,
>> I¹m not particularly knowledge on NetCDF but know that it can do 
>>parallel I/O via parallel HDF5 or ANL¹s/NU's pNetCDF? What would be 
>>the pros and cons of each configuration?
>> 
>
>The HDF5 backend ("new netcdf") allows for some nice features:  VLEN 
>arrays, compression, multiple dimensions of NC_UNLIMITED.  Those 
>features come at some cost of metadata.

Note that compression can¹t be used in HDF5 backend if doing parallel io.

ŠGreg

>
>ANL/Northwestern (thank you for mentioning both institutions!) pnetcdf 
>implements the much simpler classic NetCDF format (CDF-1, CDF-2 and 
>CDF-5), and takes advantage of the older, more restrictive constraints.
>
>If you have very large datasets, you're unlikely to see much difference 
>between the two approaches, as data movement costs will dominate.
>
>One could construct datasets impossible to implement in ANL/NU pnetcdf, 
>and one could likewise construct pathological datasets (e.g. a thousand 
>datasets, each with 4k of data in them) that would perform 
>exceptionally poorly under Unidada NetCDF.
>
>Here's a fun game you can play:  let's say you've got a representative 
>benchmark that shows Unidata NetCDF outperforming ANL/Northwestern 
>pnetcdf.  Wei-keng and I will defend our professional pride and tune 
>the heck out of pnetcdf to meet or beat our good-natured competitor.
>Likewise, Ward and team would do the same if the results were reversed.
>You can get decades worth of experience looking at your workload for 
>free!
>
>=rob
>
>> Thanks,
>> Sean
>> _______________________________________________
>> netcdfgroup mailing list
>> netcdfgroup@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe,  visit:
>>http://www.unidata.ucar.edu/mailing_lists/
>
>
>_______________________________________________
>netcdfgroup mailing list
>netcdfgroup@xxxxxxxxxxxxxxxx
>For list information or to unsubscribe,  visit:
>http://www.unidata.ucar.edu/mailing_lists/

_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/
  • 2016 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: