Re: performance degrades with filesize

• To: Ethan Alpert <ethan@xxxxxxxxxxxx>
• Subject: Re: performance degrades with filesize
• From: Russ Rew <russ@xxxxxxxxxxxxxxxx>
• Date: Mon, 10 Sep 2001 14:36:02 -0600

Ethan and John,

If writing data in the 5000-th record takes significantly longer than
writing data in the 5-th record, that would be a bug.

The way netCDF is intended to work, whenever you first extend a netCDF
file by writing more records beyond the end of the file, fill values
are written for intervening records.  This can be time consuming, but
these fill values should only be written once.  The time required for
subsequent writes at the end of the file should not depend on the size
of the file, but just on the amount of data to be appended.

I've just constructed a small example to try to duplicate the problem,
but the example seems to show things are working as intended.  In
particular the time to write the 20001-th record in an 800 Mbyte netCDF
file with 10000 floating-point values in each record is not
significantly different from the time required to write the first
record, about .01 sec on my desktop workstation:

  $ time ./wrec 20001; ls -l bench.nc

  real  0m0.01s
  user  0m0.01s
  sys   0m0.00s
  -rw-r--r--   1 russ     ustaff   800080112 Sep 10 14:21 bench.nc

I can provide the example program to test this on your system, if
you're interested.  I'm using nc_put_vara_float(), but I don't think
this should depend on whether you're using the netCDF 2 or netCDF 3
interface.

There still may be a bug under some other combination of circumstances
that is causing an anomalous performance problem, but we'll need to be
able to duplicate it here to determine what the problem is.

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
russ@xxxxxxxxxxxxxxxx                     http://www.unidata.ucar.edu

• Follow-Ups:
  • Re: performance degrades with filesize
    • From: John Galbraith

• References:
  • Re: performance degrades with filesize
    • From: Ethan Alpert