NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

Re: [netcdf-java] Errors reading certain NetCDF4 data

  • To: Antonio Rodriges <antonio.rrz@xxxxxxxxx>
  • Subject: Re: [netcdf-java] Errors reading certain NetCDF4 data
  • From: Ryan May <rmay@xxxxxxxx>
  • Date: Thu, 19 Feb 2015 11:24:36 -0700
Antonio,

Even with that chunk size, the number of bytes between consecutive points
in time is 512 x 4 x sizeof(float), which is 8 kb. You may get a few points
closer together, but they're still not close together. Any read ahead
function of the disk will be throwing away 99% of the data if all you want
is all the time for a single point.

If you're predominant access pattern is all times for a single point, your
best throughput will be achieved by making sure that those points are
consecutive on disk, which means that you should have time be the first
dimension, not the last. Anything else you do will be papering over the
core problem.

Ryan

On Thu, Feb 19, 2015 at 10:37 AM, Antonio Rodriges <antonio.rrz@xxxxxxxxx>
wrote:

> Christian,
>
> According to Russ Rew
>
> http://www.unidata.ucar.edu/blogs/developer/entry/chunking_data_why_it_matters
> the chunking must help for my access pattern
>
> After rechunking I expected to have chunks with 512x4x4 sizes where
> values for the single point and different time should be stored very
> close on disk
>



-- 
Ryan May
Software Engineer
UCAR/Unidata
Boulder, CO