Re: Read some elements of an array with NetCDF

• To: stephane.malledant@xxxxxxxx
• Subject: Re: Read some elements of an array with NetCDF
• From: Russ Rew <russ@xxxxxxxxxxxxxxxx>
• Date: Wed, 25 Apr 2001 10:00:15 -0600

Hi Stéphane,

> I'm writting some code using NetCDF format 
> My problem is that my data structure is one array (written in a NetCDF file)
> and I need to have just few elements of this array 
> I made some tests and the result is that it's faster to read the whole array
> and use some elements than read directly some elements of this array 
> (but the memory place is not the same) 
> Here it is the commands used to read : 
>       status = NF_GET_VARM_DOUBLE(IdFichier,4,start,count,stride,imap,Val)
> 
>       status = NF_GET_VAR_DOUBLE(IdFichier,4,Val) 
> I would like to know if my tests results are corrects. 
> I was thinking that the NetCDF format would allow to read few elements
> faster than read the whole array ! 

The call 

        status = NF_GET_VAR_DOUBLE(IdFichier,4,Val) 

doesn't just get one value, it reads in the whole array.  Perhaps this
was a typo, and you meant

        status = NF_GET_VAR1_DOUBLE(IdFichier,4,Index,Val) 

which would get just the value at the specified Index.

Whether this would be faster depends on the size of the array and how
close the individual array elements are in the external dataset.  The
performance of netCDF I/O in this respect is a lot like that of the
read() system call, which is what the netCDF library ultimately uses.
When you call read() to access some data on the disk, the operating
system typically reads the data in blocks and buffers it in memory.
That's why it typically takes the same amount of time to read one
value as to read a whole disk block worth of values, because in either
case a disk block must be read.  Multiple levels of caching in the
disk and memory make the situation more complicated, and can frustrate
efforts to measure performance involving small amounts of disk I/O.

> And does it exist any (other) command to read faster only some elements of
> an array in a NetCDF file ? 

If you are only accessing individual values that are a very small
fraction of all the values in a large array and they aren't
consecutive or located at regular strides along the dimensions of the
array, you can't do much better than reading one value per call, with
NF_GET_VAR_DOUBLE1(), for example.

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
russ@xxxxxxxxxxxxxxxx                     http://www.unidata.ucar.edu

• References:
  • Read some elements of an array with NetCDF
    • From: MALLEDANT Stéphane