[netcdfgroup] slow reads in 4.4.1.1 vs 4.1.3 for some files

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: [netcdfgroup] slow reads in 4.4.1.1 vs 4.1.3 for some files
• From: "Simon (Vsevolod) Ilyushchenko" <simonf@xxxxxxxxxx>
• Date: Mon, 12 Dec 2016 22:24:11 -0800

Hi,

Two of several .nc file types that I have are read ~60 times slower with
'ncdump > /dev/null' in 4.4.1.1 as compared to my older installation of
4.1.3 compiled on Linux x86_64 with more or less the same config (the
configs not identical because there are new settings). The ncdump -d
outputs are below (the files are from HYCOM and Sentinel 3).

I can do binary search over versions to narrow this down, but wanted to ask
here first if there is a known issue.

Thanks,
Simon

HYCOM:

netcdf a {
dimensions:
        depth = 2 ;
        lat = 2001 ;
        lon = 4500 ;
        time = UNLIMITED ; // (1 currently)
variables:
        double depth(depth) ;
                depth:long_name = "Depth" ;
                depth:standard_name = "depth" ;
                depth:units = "m" ;
                depth:positive = "down" ;
                depth:axis = "Z" ;
                depth:NAVO_code = 5 ;
        double lat(lat) ;
                lat:long_name = "Latitude" ;
                lat:standard_name = "latitude" ;
                lat:units = "degrees_north" ;
                lat:point_spacing = "even" ;
                lat:axis = "Y" ;
                lat:NAVO_code = 1 ;
        double lon(lon) ;
                lon:long_name = "Longitude" ;
                lon:standard_name = "longitude" ;
                lon:units = "degrees_east" ;
                lon:modulo = "360 degrees" ;
                lon:axis = "X" ;
                lon:NAVO_code = 2 ;
        double time(time) ;
                time:long_name = "Valid Time" ;
                time:units = "hours since 2000-01-01 00:00:00" ;
                time:time_origin = "2000-01-01 00:00:00" ;
                time:calendar = "gregorian" ;
                time:axis = "T" ;
                time:NAVO_code = 13 ;
        short water_u(time, depth, lat, lon) ;
                water_u:long_name = "Eastward Water Velocity" ;
                water_u:standard_name = "eastward_sea_water_velocity" ;
                water_u:units = "m/s" ;
                water_u:_FillValue = -30000s ;
                water_u:missing_value = -30000s ;
                water_u:scale_factor = 0.001f ;
                water_u:add_offset = 0.f ;
                water_u:NAVO_code = 17 ;
        short water_v(time, depth, lat, lon) ;
                water_v:long_name = "Northward Water Velocity" ;
                water_v:standard_name = "northward_sea_water_velocity" ;
                water_v:units = "m/s" ;
                water_v:_FillValue = -30000s ;
                water_v:missing_value = -30000s ;
                water_v:scale_factor = 0.001f ;
                water_v:add_offset = 0.f ;
                water_v:NAVO_code = 18 ;

// global attributes:
                :classification_level = "UNCLASSIFIED" ;
                :distribution_statement = "Approved for public release.
Distribution unlimited." ;
                :downgrade_date = "not applicable" ;
                :classification_authority = "not applicable" ;
                :institution = "Naval Oceanographic Office" ;
                :source = "HYCOM archive file" ;
                :history = "Fri Nov 20 10:01:58 2015: ncks -4 --deflate 9
-v water_u,water_v -d depth,1,2,1 hycom_glb_911_2015110200_t000_uv3z.nc
hycom_glb_911_2015110200_t000_uv3z-test.nc\narchv2ncdf3z" ;
                :field_type = "instantaneous" ;
                :Conventions = "CF-1.0 NAVO_netcdf_v1.0" ;
                :NCO = "4.4.2" ;
}

***

Sentinel 3:

dimensions:
        columns = 4865 ;
        rows = 3682 ;
variables:
        ushort Oa20_radiance(rows, columns) ;
                Oa20_radiance:_FillValue = -1US ;
                Oa20_radiance:add_offset = 0.f ;
                Oa20_radiance:ancillary_variables = "Oa20_radiance_err" ;
                Oa20_radiance:coordinates = "time_stamp altitude latitude
longitude" ;
                Oa20_radiance:long_name = "TOA radiance for OLCI
acquisition band Oa20" ;
                Oa20_radiance:scale_factor = 0.00326378f ;
                Oa20_radiance:standard_name =
"toa_upwelling_spectral_radiance" ;
                Oa20_radiance:units = "mW.m-2.sr-1.nm-1" ;
                Oa20_radiance:valid_max = -2US ;
                Oa20_radiance:valid_min = 0US ;

// global attributes:
                :absolute_orbit_number = 3517U ;
                :ac_subsampling_factor = 64US ;
                :al_subsampling_factor = 1US ;
                :comment = " " ;
                :contact = "EOHElp@xxxxxxx" ;
                :creation_time = "2016-10-20T13:00:25Z" ;
                :institution = "SVL" ;
                :netCDF_version = "4.2 of Jul  5 2012 17:07:43 $" ;
                :product_name =
"S3A_OL_1_EFR____20161020T102624_20161020T102906_20161020T130025_0162_010_079_4319_SVL_O_NR_002.SEN3"
;
                :references = "S3IPF PDS 004 - i2r1 - Product Data Format
Specification - OLCI, S3IPF PDS 002 - i1r6 - Product Data Format
Specification - Product Structures, S3IPF DPM 002 - i2r1 - Detailed
Processing Model - OLCI Level 1" ;
                :resolution = "[ 270 294 ]" ;
                :source = "IPF-OL-1-EO 06.04" ;
                :start_time = "2016-10-20T10:26:24.238891Z" ;
                :stop_time = "2016-10-20T10:29:06.207560Z" ;
                :title = "OLCI Level 1b Product, Radiance Oa20 Data Set" ;
data:

• Follow-Ups:
  • Re: [netcdfgroup] slow reads in 4.4.1.1 vs 4.1.3 for some files
    • From: Elizabeth A. Fischer