Re: [netcdfgroup] File with large number of variables

• To: Ed Hartnett <ed@xxxxxxxxxxxxxxxx>
• Subject: Re: [netcdfgroup] File with large number of variables
• From: Dani <pressec@xxxxxxxxx>
• Date: Mon, 3 May 2010 11:41:00 +0200

Setting the cache to 0 has solved the problem on the definition of the
file. Thanks a lot.

Unfortunately, I'm still not able to write efficiently to the file I
just created. It looks like every call to nc_put_vara takes memory
that is not released.

I attach a code snippet to illustrate this. It is very clear when
executing with num_var = 100 (makes the test faster),
num_elements_var=10000 and buffer_size=1.
If I increase buffer_size the problem is less obvious but it's still
there (set buffer_size = 10 and increase num_elemements_var=100000).
Does not seem to be related to num_var this time but the number of
times nc_put_vara is called.

Any ideas?

Thanks in advance,

Dani


On Fri, Apr 30, 2010 at 8:26 PM, Ed Hartnett <ed@xxxxxxxxxxxxxxxx> wrote:
> Dani <pressec@xxxxxxxxx> writes:
>
>> Hi,
>> I have to write and read data to/from a netcdf file that has 750
>> variables, all of them using unlimited dimensions (only one per
>> variable, some dimensions shared) and 10 fixed dimensions.
>>
>> I have use netcdf-4 (because of the multiple unlimited dimensions
>> requirement) and C API.
>>
>> I'm making some prototyping on my development machine (Linux 2GB RAM)
>> and found several performance issues that I hope someone can help me
>> fix/understand:
>>
>> (1) when i create a file and try to define 1000 variables (all int)
>> and a single shared unlimited dimension, the process takes all
>> available RAM (swap included) and fails with "Error (data:def closed)
>> -- HDF error" after a (long)while.
>>
>> If I do the same closing and opening the file again every 10 or 100
>> new definitions, it works fine.  I can bypass this by creating the
>> file once (ncgen) and using a copy of it on every new file, but I
>> would prefer not to. Why does creating the variables take that much
>> memory?
>
> When you create a netCDF variable, HDF5 allocates a buffer for that
> variable. The default size of the buffer is 1 MB.
>
> I have reproduced your problem, but it can be solved be explicitly
> setting the buffer size for each variable to a lower value. I have
> checked in my tests in libsrc4/tst_vars3.c, but here's the part with the
> cache setting:
>
>      for (v = 0; v < NUM_VARS; v++)
>      {
>         sprintf(var_name, "var_%d", v);
>         if (nc_def_var(ncid, var_name, NC_INT, 1, &dimid, &varid)) ERR_RET;
>         if (nc_set_var_chunk_cache(ncid, varid, 0, 0, 0.75)) ERR_RET;
>      }
>
> Note the call to nc_set_var_chunk_cache(), right after the call to
> nc_def_var.
>
> When I take this line out, I get a serious slowdown around 4000
> variables. (I have more memory available than you do.)
>
> But when I add the call to set_var_chunk_cache(), setting the chunk
> cache to zero, then there is no slowdown, even for 10,000 variables.
>
> Thanks,
>
> Ed
> --
> Ed Hartnett  -- ed@xxxxxxxxxxxxxxxx
>

  void tick(char* data) {
    clock_t c2 = clock();
    double millis = ((c2 - c1)* 1000)/ CLOCKS_PER_SEC;
    printf("%s - elapsed %f ms \n", data, millis);
    c1 = c2;
  }

  void testNetCDFLimits() {

    int num_var = 100;
    int num_elements_var = 100000;
    size_t buffer_size = 1;

    int ncid, udim;
    int varids[num_var];
    size_t start;
    int buffer[buffer_size];
    char varname[10];
    char filename[100];

    sprintf(filename, "%d-test.nc4", num_var);

    // create the file //

    if ( nc_create(filename, NC_CLOBBER | NC_NETCDF4, &ncid) ) NCERR;
    tick("created");

    if ( nc_def_dim(ncid, "udim1", 0, &udim) ) NCERR;
    if ( nc_def_dim(ncid, "udim2", 0, &udim) ) NCERR;
    tick("dimensions defined");


    for (int j = 0; j < num_var; j++) {
        sprintf(varname,"var-%d",j);
        if ( nc_def_var(ncid, varname, NC_INT, 1, &udim, &varids[j]) ) NCERR;
        if ( nc_set_var_chunk_cache(ncid, varids[j], 0, 0, 0.75) ) NCERR;
    }
    tick("variables defined");

    if( nc_enddef(ncid) ) NCERR;
    tick("endef");
    if ( nc_close(ncid) ) NCERR;
    tick("closed");


    // open for writing //
    if(nc_open(filename, NC_WRITE, &ncid)) NCERR;

    tick("opened");

    for (int j = 0; j < num_var; j++) {
      sprintf(varname,"var-%d",j);
      if ( nc_inq_varid(ncid, varname, &varids[j]) ) NCERR;
      if ( nc_set_var_chunk_cache(ncid, varids[j], 0, 0, 0.75) ) NCERR;
    }
    tick("inquired variables");

    // iterate to write on vars. On every loop buffer_size elements are written 
on all variables //
    char debug[100];
    for (int k = 0; k < num_elements_var; k = k + buffer_size) {

      for (int j = 0; j < num_var; j++) {

        for (int l = 0; l < buffer_size; l++) {
          buffer[l] = l * j;
        }
        start = k;
        if ( nc_put_vara(ncid, varids[j], &start, &buffer_size, buffer) ) NCERR;
      }
      sprintf(debug, "%d", k);
      tick( debug );
    }
    tick("variables written");

    if( nc_close(ncid) ) NCERR;
    tick("closed");



    // open for reading //
    if ( nc_open(filename, NC_NOWRITE, &ncid) ) NCERR;
    tick("open for reading");
    
    for (int j = 0; j < num_var; j++) {
      sprintf(varname,"var-%d",j);
      if( nc_inq_varid(ncid, varname, &varids[j]) ) NCERR;
      if ( nc_set_var_chunk_cache(ncid, varids[j], 0, 0, 0.75) ) NCERR;
    }
    tick("inquired variables");

    for (int k = 0; k < num_elements_var; k = k + buffer_size) {
      for (int j = 0; j < num_var; j++) {
        start = k;
        if ( nc_get_vara(ncid, varids[j], &start, &buffer_size, buffer) ) NCERR;
      }
      sprintf(debug, "%d", k);
      tick( debug );
    }
    
    tick("variables read");
    if( nc_close(ncid) ) NCERR;
    tick("closed");

}

• Follow-Ups:
  • Re: [netcdfgroup] File with large number of variables
    • From: Shibaev, Sergei

• References:
  • [netcdfgroup] File with large number of variables
    • From: Dani
  • Re: [netcdfgroup] File with large number of variables
    • From: Ed Hartnett