Re: performance degrades with filesize

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: Re: performance degrades with filesize
• From: Steve Emmerson <steve@xxxxxxxxxxxxxxxx>
• Date: Tue, 11 Sep 2001 15:55:24 -0600

Konrad,

>Date: Tue, 11 Sep 2001 12:13:45 +0200
>From: Konrad Hinsen <hinsen@xxxxxxxxxxxxxxx>
>To: noon@xxxxxxxxxxxxxxxxxxxx
>Subject: Re: performance degrades with filesize

The above message contained the following:

> I think I found what slows it down. In my Python interface, every
> write operation to an array with an unlimited dimension is followed by
> a call to nc_inq_dimlen() in order to keep the internal dimension
> count up to date. I had assumed that this call would be cheap and
> independent of the file size, since it ought to be no more than an
> access to some data structure that the netCDF library should keep in
> memory.

I ported a previously-posted Python script to C and tested it. There was
no degradation in performance. I included calls to nc_inq_dimlen().  The
program is enclosed.

I suspect the problem lies in the Python interface.

Regards,
Steve Emmerson   <http://www.unidata.ucar.edu>

#include <netcdf.h>
#include <time.h>
#include <stdio.h>

#define X0_SIZE 10
#define X1_SIZE 50

main()
{
    int   ncId;
    int   x0Id, x1Id, x2Id, x3Id, x4Id, timeId;

    nc_create("garbage.nc", NC_CLOBBER, &ncId);
    nc_def_dim(ncId, "x0", X0_SIZE, &x0Id);
    nc_def_dim(ncId, "x1", X1_SIZE, &x1Id);
    nc_def_dim(ncId, "x2", 23, &x2Id);
    nc_def_dim(ncId, "x3", 15, &x3Id);
    nc_def_dim(ncId, "x4", 125, &x4Id);
    nc_def_dim(ncId, "time", NC_UNLIMITED, &timeId);

    {
        int y0Id, y1Id;
        int dimIds[3];
        dimIds[0] = timeId;
        dimIds[1] = x0Id;
        dimIds[2] = x1Id;
        nc_def_var(ncId, "y0", NC_FLOAT, 3, dimIds, &y0Id);
        nc_def_var(ncId, "y1", NC_FLOAT, 1, dimIds, &y1Id);
        nc_enddef(ncId);
        {
            float   y0[X1_SIZE*X0_SIZE];
            float   y1[1];
            size_t  y0Start[] = {0, 0, 0};
            size_t  y0Count[] = {1, X0_SIZE, X1_SIZE};
            size_t  y1Start[] = {0};
            size_t  y1Count[] = {1};
            clock_t oldClock = clock();
            int     i, j, time = 0;
            for (i = 0; i < 10; i++)
            {
                for (j = 0; j < 100; j++)
                {
                    size_t x0Len, x1Len, timeLen;
                    y0Start[0] = y1Start[0] = time++;
                    nc_put_vara_float(ncId, y0Id, y0Start, y0Count, y0);
                    {
                        nc_inq_dimlen(ncId, x0Id, &x0Len);
                        nc_inq_dimlen(ncId, x1Id, &x1Len);
                        nc_inq_dimlen(ncId, timeId, &timeLen);
                    }
                    nc_put_vara_float(ncId, y1Id, y1Start, y1Count, y1);
                    {
                        nc_inq_dimlen(ncId, x0Id, &x0Len);
                        nc_inq_dimlen(ncId, x1Id, &x1Len);
                        nc_inq_dimlen(ncId, timeId, &timeLen);
                    }
                }
                {
                    clock_t newClock = clock();
                    printf(
                        "%f\n", (newClock - oldClock)/(double)CLOCKS_PER_SEC);
                    oldClock = newClock;
                }
            }
            nc_close(ncId);
        }
    }
}

• References:
  • Re: performance degrades with filesize
    • From: Konrad Hinsen