[netcdfgroup] Slow nc_get_vara_ when operating on NetCDF-4 files [uncl]

• To: "netcdfgroup@xxxxxxxxxxxxxxxx" <netcdfgroup@xxxxxxxxxxxxxxxx>
• Subject: [netcdfgroup] Slow nc_get_vara_ when operating on NetCDF-4 files [uncl]
• From: Timothy Hume <T.Hume@xxxxxxxxxx>
• Date: Thu, 12 May 2011 12:47:06 +1000

Hi,

I have encountered a situation where nc_get_vara and nc_get_vars operations run 
much slower on NetCDF-4 files than they do on NetCDF-3 files.

Here is an example CDL for one of my files (with irrelevant stuff removed):

netcdf ecmwf_0p5deg_ocf_t2m_mars_opn {
dimensions:
        basetime = UNLIMITED ; // (100 currently)
        forecast = 32 ;
        bounds = 2 ;
        latitude = 121 ;
        longitude = 101 ;
variables:
        
        float temperature_2m(basetime, forecast, latitude, longitude) ;
                temperature_2m:long_name = "Air temperature 2m above the 
surface" ;
                temperature_2m:units = "K" ;
                temperature_2m:_FillValue = 9.96921e+36f ;
                temperature_2m:missing_value = 9.96921e+36f ;
                temperature_2m:valid_min = 180.f ;
                temperature_2m:valid_max = 330.f ;
                temperature_2m:standard_name = "air_temperature" ;
                temperature_2m:cell_methods = "area: mean" ;
                temperature_2m:coordinates = "level" ;
}

I can create a NetCDF-3 or NetCDF-4 file using this CDL. The problem is that 
when I try slicing out a single record (on the unlimited dimension) using 
nc_get_vara or nc_get_vars functions, it can take an order of magnitude 
(really) longer on a NetCDF-4 file than with a NetCDF-3 file. This affects 
NetCDF-4 files with or without compression. I've experimented with various 
NetCDF-4 chunking options, and that doesn't help either.

Has anyone else encountered such slowness issues with NetCDF-4? My solution is 
to keep the data files in NetCDF-3, but I would really like to migrate to 
NetCDF-4.

Tim Hume
Centre for Australian Weather and Climate Research
Australian Bureau of Meteorology

• Follow-Ups:
  • Re: [netcdfgroup] Slow nc_get_vara_ when operating on NetCDF-4 files [uncl]
    • From: Ed Hartnett