NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
I would like to add one feature of PnetCDF that it supports I/O request aggregation through PnetCDF nonblocking APIs. This feature can combine multiple (small) requests into large one, so to achieve a better performance. This is particularly useful if you have many variables to write and/or read. Wei-keng On Mar 2, 2016, at 4:25 PM, dmh@xxxxxxxx wrote: > That is correct. Pnetcdf only supports the netcdf-3 format and it offshoot > CDF5. They do not support chunking or compression. > =Dennis Heimbigner > Unidata > > On 3/2/2016 3:01 PM, Kent Yang wrote: >> I don't think pnetcdf from ANL uses the chunking technique as the HDF5 does. >> That may lead to bigger performance difference when some subset patterns get >> involved. >> >> Kent >> >> -----Original Message----- >> From: netcdfgroup-bounces@xxxxxxxxxxxxxxxx >> [mailto:netcdfgroup-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Sjaardema, >> Gregory D >> Sent: Wednesday, March 02, 2016 3:55 PM >> To: Latham, Robert J.; seanb@xxxxxxxx >> Cc: netcdfgroup@xxxxxxxxxxxxxxxx >> Subject: Re: [netcdfgroup] [EXTERNAL] Re: NetCDF parallel I/O configurations >> >> >> On 3/2/16, 2:49 PM, "netcdfgroup-bounces@xxxxxxxxxxxxxxxx on behalf of >> Latham, Robert J." <netcdfgroup-bounces@xxxxxxxxxxxxxxxx on behalf of >> robl@xxxxxxxxxxx> wrote: >> >>> On Tue, 2016-02-23 at 17:13 +0000, Sean Byland wrote: >>>> Hello, >>>> I¹m not particularly knowledge on NetCDF but know that it can do >>>> parallel I/O via parallel HDF5 or ANL¹s/NU's pNetCDF? What would be >>>> the pros and cons of each configuration? >>>> >>> The HDF5 backend ("new netcdf") allows for some nice features: VLEN >>> arrays, compression, multiple dimensions of NC_UNLIMITED. Those >>> features come at some cost of metadata. >> Note that compression can¹t be used in HDF5 backend if doing parallel io. >> >> ŠGreg >> >>> ANL/Northwestern (thank you for mentioning both institutions!) pnetcdf >>> implements the much simpler classic NetCDF format (CDF-1, CDF-2 and >>> CDF-5), and takes advantage of the older, more restrictive constraints. >>> >>> If you have very large datasets, you're unlikely to see much difference >>> between the two approaches, as data movement costs will dominate. >>> >>> One could construct datasets impossible to implement in ANL/NU pnetcdf, >>> and one could likewise construct pathological datasets (e.g. a thousand >>> datasets, each with 4k of data in them) that would perform >>> exceptionally poorly under Unidada NetCDF. >>> >>> Here's a fun game you can play: let's say you've got a representative >>> benchmark that shows Unidata NetCDF outperforming ANL/Northwestern >>> pnetcdf. Wei-keng and I will defend our professional pride and tune >>> the heck out of pnetcdf to meet or beat our good-natured competitor. >>> Likewise, Ward and team would do the same if the results were reversed. >>> You can get decades worth of experience looking at your workload for >>> free! >>> >>> =rob >>> >>>> Thanks, >>>> Sean >>>> _______________________________________________ >>>> netcdfgroup mailing list >>>> netcdfgroup@xxxxxxxxxxxxxxxx >>>> For list information or to unsubscribe, visit: >>>> http://www.unidata.ucar.edu/mailing_lists/ >>> >>> _______________________________________________ >>> netcdfgroup mailing list >>> netcdfgroup@xxxxxxxxxxxxxxxx >>> For list information or to unsubscribe, visit: >>> http://www.unidata.ucar.edu/mailing_lists/ >> _______________________________________________ >> netcdfgroup mailing list >> netcdfgroup@xxxxxxxxxxxxxxxx >> For list information or to unsubscribe, visit: >> http://www.unidata.ucar.edu/mailing_lists/ >> >> _______________________________________________ >> netcdfgroup mailing list >> netcdfgroup@xxxxxxxxxxxxxxxx >> For list information or to unsubscribe, visit: >> http://www.unidata.ucar.edu/mailing_lists/ > > _______________________________________________ > netcdfgroup mailing list > netcdfgroup@xxxxxxxxxxxxxxxx > For list information or to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/
netcdfgroup archives: