Re: [netcdf-java] Writing a 2 Gb NetCDF File

• To: Valerio Angelini <angelini@xxxxxxxxxxx>
• Subject: Re: [netcdf-java] Writing a 2 Gb NetCDF File
• From: John Caron <caron@xxxxxxxxxxxxxxxx>
• Date: Tue, 17 Jun 2008 12:16:50 -0600

Hi Valerio:

1. 2.2 does not support writing large file.

    ncFile.setLargeFile(true);

to enable files > 2 Gb.

3. for performance, call

    ncFile.setFill(false);
    ncFile.setLength(approxSize);

before the create() call. with this, it took 267 secs to run on my windows 
machine.

thanks for helping to debug!

-----

  public void testBig() throws IOException, InvalidRangeException {

    long start = System.nanoTime();
    System.out.println("Begin <=");

    String varName = "example";

    int timeSize = 8;
    int latSize = 8022;
    int lonSize = 10627;

    ncFile.setFill(false);
    ncFile.setLargeFile(true);

    long approxSize = (long) timeSize * latSize * lonSize * 4 + 4000;
    ncFile.setLength(approxSize);

    String timeUnits = "hours since 2008-06-06 12:00:0.0";
    String coordUnits = "degrees";

    Dimension[] dim = new Dimension[3];

    dim[0] = setDimension(ncFile, "time", timeUnits, timeSize);
    dim[1] = setDimension(ncFile, "lat", coordUnits, latSize);
    dim[2] = setDimension(ncFile, "lon", coordUnits, lonSize);

    ncFile.addVariable(varName, DataType.FLOAT, dim);

    ncFile.addVariableAttribute(varName, "_FillValue", -9999);
    ncFile.addVariableAttribute(varName, "missing_value", -9999);

    System.out.println("Creating netcdf <=");
    ncFile.create();
    long stop = System.nanoTime();
    double took = (stop - start) * .001 * .001 * .001;
    System.out.println("That took "+took+" secs");
    start = stop;

    System.out.println("Writing netcdf <=");

    int[] shape = new int[]{1, 1, lonSize};
    float[] floatStorage = new float[lonSize];
    Array floatArray = Array.factory(float.class, shape, floatStorage);
    for (int t = 0; t < timeSize; t++) {
      for (int i = 0; i < latSize; i++) {
        int[] origin = new int[]{t, i, 0};
        ncFile.write(varName, origin, floatArray);
      }
    }

    ncFile.close();

    System.out.println("Done <=");
    stop = System.nanoTime();
    took = (stop - start) * .001 * .001 * .001;
    System.out.println("That took "+took+" secs");
    start = stop;
  }

    Dimension dimension = ncFile.addDimension(name, length);
    ncFile.addVariable(name, DataType.FLOAT, new Dimension[]{dimension});
    ncFile.addVariableAttribute(name, "units", units);

    return dimension;
  }

• References:
  • [netcdf-java] Writing a 2 Gb NetCDF File
    • From: Valerio Angelini