NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

Re: [netcdfgroup] slow reads in 4.4.1.1 vs 4.1.3 for some files

  • To: Dave Allured - NOAA Affiliate <dave.allured@xxxxxxxx>
  • Subject: Re: [netcdfgroup] slow reads in 4.4.1.1 vs 4.1.3 for some files
  • From: Chris Barker <chris.barker@xxxxxxxx>
  • Date: Thu, 15 Dec 2016 11:17:15 -0800
On Wed, Dec 14, 2016 at 9:13 PM, Dave Allured - NOAA Affiliate <
dave.allured@xxxxxxxx> wrote:


> So I think you have a read cacheing failure, due to interaction between
> the ncdump read pattern, and your chunking scheme.
>
...

> A sampling tool found that ncdump was spending more than 96% of its time
> inside an HDF5 chunk reader with decompression.  Every time an HDF5 chunk
> is physically read from disk, the *entire* chunk must be decompressed, even
> to access a single value.  You see why chunk cacheing is important.
>

Does HDF5 not do any chunk caching itself? or for that matter, netcdf4? Is
is really up to the application level to manage the caching? that seems
like handling it at the wrong level to me.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker@xxxxxxxx