Re: [netcdfgroup] NetCDF4/HDF5 File Single Variable Read Times

• To: john.storrs@xxxxxxxxxxxx
• Subject: Re: [netcdfgroup] NetCDF4/HDF5 File Single Variable Read Times
• From: Russ Rew <russ@xxxxxxxxxxxxxxxx>
• Date: Fri, 04 Sep 2009 10:16:52 -0600

Hi John,

> We've noticed a large difference in single variable read times between 
> netCDF-4/HDF5 reads made with the netCDF-4 API (slower) and reads made with 
> the HDF5 API (faster). Single variable reads are a common access pattern in 
> fusion research, where 1000s of files are often scanned to analyse 
> experimental data for a particular variable. 
> 
> This came to light when benchmarking reads of a netCDF-4/HDF5 file with 3000 
> variables, where a single variable read (open, read, close) took 11ms using 
> the HDF5 API and 1300ms using the netCDF-4 API. In contrast, multiple 
> variable reads of the open file with the netCDF-4 API took 0.7ms each. 
> Evidently the netCDF-4 API builds an internal data structure upfront to 
> assist possible later access, while the HDF5 API postpones that until access 
> is actually needed.
> 
> We are considering using the HDF5 API for fast single variable access. Is 
> there any other way to avoid the netCDF-4 file open overhead?

Can you make a sample file available?  We might be able to speed up
opens or provide a flag for fast opens that specifies not reading all
the metadata initially.

Thanks!

--Russ
_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
russ@xxxxxxxxxxxxxxxx                     http://www.unidata.ucar.edu

• References:
  • [netcdfgroup] NetCDF4/HDF5 File Single Variable Read Times
    • From: John Storrs