Re: Performance problem with large files

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: Re: Performance problem with large files
• From: Martin Dix <mrd@xxxxxxxxxxxx>
• Date: Mon, 28 Jun 1999 09:48:39 +1000 (AEST)

hinsen@xxxxxxxxxxxxxxxxxxxxx writes:

 > ... The data in the files is essentially
 > one single-precision float array of dimensions 8000 x 3 x 16000, the
 > last dimension being declared as "unlimited". I read and write
 > subarrays of shape 1 x 3 x 16000. ...

For simplicity call the unlimited dimension t. A netcdf file stores
all the data for t=1, then for t=2 etc. Your description of the
array indices means that each subarray is scattered through the
entire file and requires accessing almost every file block. Things
should be a lot better if you write subarrays of 8000 x 3 x 1 or if
you can't do this, rearrange the file so that the 8000 dimension is
unlimited rather than the 16000 dimension.

Martin Dix 

CSIRO Atmospheric Research                Phone: +61 3 9239 4533
Private Bag No. 1, Aspendale                Fax: +61 3 9239 4444
Victoria 3195  Australia                  Email: martin.dix@xxxxxxxxxxxx