[netcdfgroup] Problem to read slices from chunked and compressed data

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: [netcdfgroup] Problem to read slices from chunked and compressed data
• From: Wei Huang <huangwei@xxxxxxxx>
• Date: Wed, 1 Feb 2012 15:03:06 -0700

NetCDF developers,

Recently the NCL group got a dataset from user which cause
segmentation fault.

Here are the some info about this dataset (from out ncl script):

> cat b.ncl 
 setfileoption("nc", "usenewhlfs", True)

 f=addfile("vp.850.nc","r")
 print(f)


> ncl b.ncl
 Copyright (C) 1995-2012 - All Rights Reserved
 University Corporation for Atmospheric Research
 NCAR Command Language Version 6.1.0
 The use of this software is governed by a License Agreement.
 See http://www.ncl.ucar.edu/ for more details.

Variable: f
Type: file
filename:       vp.850
path:   vp.850.nc
attributes:
    description :       ERA Interim vp at 850 hPa.
    references  :       Simmons et al. 2006
    source      :       ECMWF: 
http://data-portal.ecmwf.int/data/d/interim_daily/
    creation_date       :       Thu Jan 26 17:59:46 UTC 2012
    source_file :       /ecmwf/interim//*/u&v*.grib

dimensions:
    time        = 33349 // unlimited
    lat = 121
    lon = 240

chunk dimensions:
    time        = 1 // unlimited
    lat = 121
    lon = 240

variables:
    time: <double>      [ 33349 <time | unlimited> ]
        Chunking Info:  [ 1 <time | unlimited> ]
        attributes:
            last_time   :       2011-10-30 00:00:00
            first_time  :       1989-01-01 00:00:00
            delta_t     :       0000-00-00 06:00:00
            long_name   :       initial time
            units       :       hours since 1800-01-01 00:00:00

    lat: <float>        [ 121 <lat> ]
        Chunking Info:  [ 121 <lat> ]
        attributes:
            actual_range        :       ( -90.000000, 90.000000 )
            delta_y     :       1.500000
            coordinate_defines  :       center
            mapping     :       cylindrical_equidistant_projection_grid
              grid_resolution     :       1.5_degrees
            long_name   :       latitude
            units       :       degrees_north

    lon: <float>        [ 240 <lon> ]
        Chunking Info:  [ 240 <lon> ]
        attributes:
            actual_range        :       ( -180.000000, 178.500000 )
            delta_x     :       1.500000
            coordinate_defines  :       center
            mapping     :       cylindrical_equidistant_projection_grid
            grid_resolution     :       1.5_degrees
            long_name   :       longitude
            units       :       degrees_east

    vp: <float> [ 33349 <time | unlimited>, 121 <lat>, 240 <lon> ]
        Chunking Info:  [ 1 <time | unlimited>, 121 <lat>, 240 <lon> ]
        attributes:
            _FillValue  :       -999.900024
            mapping     :       cylindrical_equidistant_projection_grid
            grid_resolution     :       1.5_degrees
            pressure_level      :       7310034281957426002
            units       :       m^2s^-1
            long_name   :       velocity_potential

By debugging with this dataset,
we are able to trace back the problem is from:
libsrc4/nc4hdf.c, in function nc4_get_var().

We made some change to this function and now NCL works.

The code we have changed are:

start from line: 1075:

The original:

   if (!scalar && provide_fill)
   {
      void *filldata;
      int real_data_size = 0;
      int fill_len;

      /* Skip past the real data we've already read. */
      if (!no_read)
         for (real_data_size = 1, d2 = 0; d2 < var->ndims; d2++)
            real_data_size *= (count[d2] - start[d2]) * file_type_size;

to new:

   if (!scalar && provide_fill)
   {
      void *filldata;
      size_t real_data_size = 0;
      size_t fill_len;

      /* Skip past the real data we've already read. */
      if (!no_read)
      {
         for (real_data_size = 1, d2 = 0; d2 < var->ndims; d2++)
            real_data_size *= (count[d2] - start[d2]);
         real_data_size *= file_type_size;
       }


Regards,

Wei

huangwei@xxxxxxxx
VETS/CISL
National Center for Atmospheric Research
P.O. Box 3000 (1850 Table Mesa Dr.)
Boulder, CO 80307-3000 USA
(303) 497-8924