NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
That is correct. Pnetcdf only supports the netcdf-3 format and it offshoot CDF5. They do not support chunking or compression. =Dennis Heimbigner Unidata On 3/2/2016 3:01 PM, Kent Yang wrote:
I don't think pnetcdf from ANL uses the chunking technique as the HDF5 does. That may lead to bigger performance difference when some subset patterns get involved. Kent -----Original Message----- From: netcdfgroup-bounces@xxxxxxxxxxxxxxxx [mailto:netcdfgroup-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Sjaardema, Gregory D Sent: Wednesday, March 02, 2016 3:55 PM To: Latham, Robert J.; seanb@xxxxxxxx Cc: netcdfgroup@xxxxxxxxxxxxxxxx Subject: Re: [netcdfgroup] [EXTERNAL] Re: NetCDF parallel I/O configurations On 3/2/16, 2:49 PM, "netcdfgroup-bounces@xxxxxxxxxxxxxxxx on behalf of Latham, Robert J." <netcdfgroup-bounces@xxxxxxxxxxxxxxxx on behalf of robl@xxxxxxxxxxx> wrote:On Tue, 2016-02-23 at 17:13 +0000, Sean Byland wrote:Hello, I¹m not particularly knowledge on NetCDF but know that it can do parallel I/O via parallel HDF5 or ANL¹s/NU's pNetCDF? What would be the pros and cons of each configuration?The HDF5 backend ("new netcdf") allows for some nice features: VLEN arrays, compression, multiple dimensions of NC_UNLIMITED. Those features come at some cost of metadata.Note that compression can¹t be used in HDF5 backend if doing parallel io. ŠGregANL/Northwestern (thank you for mentioning both institutions!) pnetcdf implements the much simpler classic NetCDF format (CDF-1, CDF-2 and CDF-5), and takes advantage of the older, more restrictive constraints. If you have very large datasets, you're unlikely to see much difference between the two approaches, as data movement costs will dominate. One could construct datasets impossible to implement in ANL/NU pnetcdf, and one could likewise construct pathological datasets (e.g. a thousand datasets, each with 4k of data in them) that would perform exceptionally poorly under Unidada NetCDF. Here's a fun game you can play: let's say you've got a representative benchmark that shows Unidata NetCDF outperforming ANL/Northwestern pnetcdf. Wei-keng and I will defend our professional pride and tune the heck out of pnetcdf to meet or beat our good-natured competitor. Likewise, Ward and team would do the same if the results were reversed. You can get decades worth of experience looking at your workload for free! =robThanks, Sean _______________________________________________ netcdfgroup mailing list netcdfgroup@xxxxxxxxxxxxxxxx For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/_______________________________________________ netcdfgroup mailing list netcdfgroup@xxxxxxxxxxxxxxxx For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/_______________________________________________ netcdfgroup mailing list netcdfgroup@xxxxxxxxxxxxxxxx For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/ _______________________________________________ netcdfgroup mailing list netcdfgroup@xxxxxxxxxxxxxxxx For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/
netcdfgroup archives: