Re: performance degrades with filesize

• To: noon@xxxxxxxxxxxxxxxxxxxx
• Subject: Re: performance degrades with filesize
• From: Konrad Hinsen <hinsen@xxxxxxxxxxxxxxx>
• Date: Tue, 11 Sep 2001 12:13:45 +0200

> John -- I simplified and ported your test to my netcdf/python module that
> doesn't tie into Numeric the way that Konrad Hinsen's does.  I just filled
> with a constant value.
> 
> When run, I get a flat load time of < 0.01 sec/10 interations.

I think I found what slows it down. In my Python interface, every
write operation to an array with an unlimited dimension is followed by
a call to nc_inq_dimlen() in order to keep the internal dimension
count up to date. I had assumed that this call would be cheap and
independent of the file size, since it ought to be no more than an
access to some data structure that the netCDF library should keep in
memory.

Apparently this is not true, as I just removed this call
experimentally, which speeds up John's script dramatically and yields
a constant access time. Adding an explicit shape request elsewhere
slows it down again, so I am quite sure the the length inquiry causes
the problem.

I don't really understand what is going on here. Is the whole file scanned
to find the length? There must be a quicker way to do that.

Anyway, I can always substitute my own code to figure out the new
length based on the old length and the last write operation.

Konrad.
-- 
-------------------------------------------------------------------------------
Konrad Hinsen                            | E-Mail: hinsen@xxxxxxxxxxxxxxx
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
-------------------------------------------------------------------------------

• Follow-Ups:
  • Re: performance degrades with filesize
    • From: Steve Emmerson

• References:
  • Re: performance degrades with filesize
    • From: William Noon