Re: [netcdfgroup] Slow nc_get_vara_ when operating on NetCDF-4 files [uncl]

• To: Timothy Hume <T.Hume@xxxxxxxxxx>
• Subject: Re: [netcdfgroup] Slow nc_get_vara_ when operating on NetCDF-4 files [uncl]
• From: Ed Hartnett <ed@xxxxxxxxxxxxxxxx>
• Date: Thu, 12 May 2011 05:39:14 -0600

Timothy Hume <T.Hume@xxxxxxxxxx> writes:

> Hi,
>
> I have encountered a situation where nc_get_vara and nc_get_vars operations 
> run much slower on NetCDF-4 files than they do on NetCDF-3 files.
>
> Here is an example CDL for one of my files (with irrelevant stuff removed):
>
> netcdf ecmwf_0p5deg_ocf_t2m_mars_opn {
> dimensions:
>       basetime = UNLIMITED ; // (100 currently)
>       forecast = 32 ;
>       bounds = 2 ;
>       latitude = 121 ;
>       longitude = 101 ;
> variables:
>       
>       float temperature_2m(basetime, forecast, latitude, longitude) ;
>               temperature_2m:long_name = "Air temperature 2m above the 
> surface" ;
>               temperature_2m:units = "K" ;
>               temperature_2m:_FillValue = 9.96921e+36f ;
>               temperature_2m:missing_value = 9.96921e+36f ;
>               temperature_2m:valid_min = 180.f ;
>               temperature_2m:valid_max = 330.f ;
>               temperature_2m:standard_name = "air_temperature" ;
>               temperature_2m:cell_methods = "area: mean" ;
>               temperature_2m:coordinates = "level" ;
> }
>
> I can create a NetCDF-3 or NetCDF-4 file using this CDL. The problem is that 
> when I try slicing out a single record (on the unlimited dimension) using 
> nc_get_vara or nc_get_vars functions, it can take an order of magnitude 
> (really) longer on a NetCDF-4 file than with a NetCDF-3 file. This affects 
> NetCDF-4 files with or without compression. I've experimented with various 
> NetCDF-4 chunking options, and that doesn't help either.
>
> Has anyone else encountered such slowness issues with NetCDF-4? My solution 
> is to keep the data files in NetCDF-3, but I would really like to migrate to 
> NetCDF-4.
>

Howdy Tim!

Can you try this with the latest beta release?

There have been many performance improvements since the 4.1.1 release...

Thanks,

Ed

-- 
Ed Hartnett  -- ed@xxxxxxxxxxxxxxxx

• References:
  • [netcdfgroup] Slow nc_get_vara_ when operating on NetCDF-4 files [uncl]
    • From: Timothy Hume