NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Hi there We've noticed a large difference in single variable read times between netCDF-4/HDF5 reads made with the netCDF-4 API (slower) and reads made with the HDF5 API (faster). Single variable reads are a common access pattern in fusion research, where 1000s of files are often scanned to analyse experimental data for a particular variable. This came to light when benchmarking reads of a netCDF-4/HDF5 file with 3000 variables, where a single variable read (open, read, close) took 11ms using the HDF5 API and 1300ms using the netCDF-4 API. In contrast, multiple variable reads of the open file with the netCDF-4 API took 0.7ms each. Evidently the netCDF-4 API builds an internal data structure upfront to assist possible later access, while the HDF5 API postpones that until access is actually needed. We are considering using the HDF5 API for fast single variable access. Is there any other way to avoid the netCDF-4 file open overhead? Regards John -- John Storrs, Experiments Dept e-mail: john.storrs@xxxxxxxxxxxx Building D3, UKAEA Fusion tel: 01235 466338 Culham Science Centre fax: 01235 466379 Abingdon, Oxfordshire OX14 3DB http://www.fusion.org.uk This message has been scanned for viruses by BlackSpider MailControl - www.blackspider.com
netcdfgroup archives: