Re: performance degrades with filesize

• To: John Galbraith <john@xxxxxxxxxxxxxxx>
• Subject: Re: performance degrades with filesize
• From: Timothy Hume <tim.hume@xxxxxxxxx>
• Date: Mon, 10 Sep 2001 20:52:03 +0000 (UTC)

Hi,

I have also experienced the problem you describe. I am using the C
interface. I have been converting meteorological observations to NetCDF.
When a new observation comes in, I simply grow the unlimited dimension by
1. At first, when the files only contain a few observations, things go
fast, but as the files get larger, writing to them slows down.

I rewrote my code to grow the unlimited dimension only occasionally, as
you describe (I grow it in increments of 100 observations). I got a small
performance increase by doing things this way, but not nearly as large as
I was hoping for. The reason for the relatively slow large files still
remains a mystery to me (and in my case it may not be a NetCDF problem
....)

Tim Hume
Meteorological Service of New Zealand
Wellington
New Zealand

> Well, I could save growing the dimension by allocating a huge file at
> first, and then chop off the unused portion when my simulation completed.
> I could also grow the time dimension only occasionally, but not every
> timestep.  Unfortunately, I don't think that this is the whole issue.  I
> suppose that not using an unlimited dimension might be a work around, but
> it is basically a stab in the dark.  It is probably just a bug somewhere
> (it could be in my own code), because I would think that you could append
> the file fast as far as how the underlying C library works.
> 
> Thanks,        
>        John
> 
> 

• References:
  • Re: performance degrades with filesize
    • From: John Galbraith