[netcdfgroup] [netCDF #CAE-726196]: optimal chunking and cache for interpolating sparse data in a large grid

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: [netcdfgroup] [netCDF #CAE-726196]: optimal chunking and cache for interpolating sparse data in a large grid
• From: Armin Corbin <corbin@xxxxxxxxxxxxxxxx>
• Date: Wed, 22 Jul 2020 09:54:31 +0200

Dear NetCDF developers,

I have a rather specific question on chunk size and the cache size.

I have 4D data (time, height, latitude, longitude) given on a curve
linear grid. The grid is regular in time, latitude and longitude, but
the height component differs form element to element:

The actual data (for one day) is stored in:

float DEN(time=144, ilev=57, lat=72, lon=144);
      :long_name = "Total Density";
      :units = "g/cm3";
      :missing_value = 1.0E36; // double
      :_ChunkSizes = 1U, 57U, 72U, 144U; // uint

The height of each grid cell is stored in:

float ZG(time=144, ilev=57, lat=72, lon=144);
      :long_name = "Geometric Height ZG";
      :units = "cm";
      :missing_value = 1.0E36; // double
      :_ChunkSizes = 1U, 57U, 72U, 144U; // uint

I interpolate a trajectory of a satellite in this grid. Thus, I need
only a small fraction of the data.

Using linear interpolation, for each satellite position, I need to
access 8 height profiles in ZG: ZG(time_idx, :, lat_idx, lon_idx).
I call nc_get_vara_float 8 times to get all profiles. I do not use a
single call of nc_get_vara_float because the grid is periodic in
longitude and a single call would not work on the boundaries.

Afterwards, I need 16 elements from the variable DEN which I access
individually with nc_get_var1_float(ncid, varid, &indexp[0], &value);
Again I do not use nc_get_vara_float due to the boundaries.

Currently, each point in time is one chunk. This seems to be not
optimal for the described scenario. Do you have any experience with that
kind of scenario? I have tried to change the chunk size to 1x19x18x18, but
that increased the run-time.


Another aspect is the chunk cache which can be altered with:

nc_set_var_chunk_cache(ncid, varid, CACHE_SIZE, CACHE_NELEMS,
CACHE_PREEMPTION))

So the cache size should be:
elements per chunk * CACHE_NELEMS * sizeof(float)

Is that correct?

What would be a good value for CACHE_PREEMPTION ?

Thanks for your help!

Armin Corbin



--
Armin Corbin

Astronomical, Physical and Mathematical Geodesy,
Institute for Geodesy and Geoinformation
University of Bonn
Nußallee 15
53115 Bonn, Germany

Telephone: +49(0)22873-2907
E-Mail:    corbin@xxxxxxxxxxxxxxxx

• Follow-Ups:
  • Re: [netcdfgroup] [netCDF #CAE-726196]: optimal chunking and cache for interpolating sparse data in a large grid
    • From: Julian Kunkel