[netcdf-java] ArrayIndexOutOfBoundException when generating a large file

• To: <netcdf-java@xxxxxxxxxxxxxxxx>
• Subject: [netcdf-java] ArrayIndexOutOfBoundException when generating a large file
• From: <Jean-Michel.Perraud@xxxxxxxx>
• Date: Thu, 30 Apr 2009 16:27:12 +1000

Hi,

I want to generate a lat*lon*time data file which I estimate will be ~16GB. I 
am using the latitude as an unlimited dimension to (1) lay out the file for 
efficient reading of time series and (2) as I understand, overcome the 4GB size 
limit. Things work fine with small data. Testing with lat=300, lon=300 and 
time=40000 the file fails to grow beyond 8GB. This is only my second 
significant use of netcdf and I am probably missing something obvious to 
seasoned netCDF users.

I am using netcdfAll-4.0.jar. The stack trace in Eclipse is from the close() 
method, but things must go pear-shaped before as only half the records are 
output (179 out of 300 latitudes).

Any advice would be appreciated. Sample code/netcdf structure follows.
Cheers
-------------------------------------------
Jean-Michel Perraud
Prediction and Reporting Technologies, CSIRO Land and Water


Thread [main] (Suspended (exception ArrayIndexOutOfBoundsException))    
        RandomAccessFile.write(int) line: 747    // 
ArrayIndexOutOfBoundsException: -2013020
        RandomAccessFile.writeInt(int) line: 1438       
        N3header.writeNumrecs() line: 900       
        N3raf(N3iosp).flush() line: 869 
        NetcdfFileWriteable.flush() line: 731   
        NetcdfFileWriteable.close() line: 741   
        TestLargeGeneration.main(String[]) line: 55     

netcdf file://XXXX/tmp/test.nc {
 dimensions:
   time = 40000;
   lat = UNLIMITED;   // (179 currently)
   lon = 300;
 variables:
   float testVar(lat=179, lon=300, time=40000);
     :long_name = "testVar";
     :units = "mm";
     :missing_value = -9999.0f; // float
     :_FillValue = -9999.0f; // float
   char cellId(lat=179, lon=300);
     :long_name = "Cell ID";
   float lat(lat=179);
     :units = "degrees_north";
     :axis = "Y";
     :standard_name = "latitude";
     :_CoordinateAxisType = "Lat";
   float lon(lon=300);
     :units = "degrees_east";
     :axis = "X";
     :standard_name = "longitude";
     :_CoordinateAxisType = "Lon";
   int time(time=40000);
     :units = "days since 1889-01-01";
     :axis = "T";
     :standard_name = "time";
     :long_name = "time";
     :_CoordinateAxisType = "Time";
}
#################################
                try {
                        writeableFile = createLatLonTimeDataCube(filename);
                        // The outer dimension has shape 1, since we will write 
one record
                        // at a time, and one
                        // record corresponds to one latitude ordinate
                        ArrayFloat.D3 variableData = new ArrayFloat.D3(1, 
LON_LEN, TIME_LEN);
                        // ArrayObject.D2 cellIdData = new
                        // ArrayObject.D2(class(java.lang.String), 
latDim.getLength(),
                        // lonDim.getLength());
                        // An array to record the latitude ordinate
                        ucar.ma2.Array latData = 
ucar.ma2.Array.factory(DataType.FLOAT,
                                        new int[] { 1 });

                        // The origin to use to write the runoff record
                        int[] origin = new int[] { 0, 0, 0 }; // lat, lon, time
                        // The origin to use to write the latitude for each 
record
                        int[] lat_origin = new int[] { 0 };

                        for (int lat = 0; lat < LAT_LEN; lat++) {
                                latData.setFloat(latData.getIndex(), (float) 
(lat / 10.0));
                                for (int lon = 0; lon < LON_LEN; lon++) {
                                        for (int time = 0; time < TIME_LEN; 
time++) {
                                                variableData.set(0, lon, time, 
(float) time);
                                        }
                                }
                                // write the data out for this record
                                origin[0] = lat;
                                lat_origin[0] = lat;
                                writeableFile.write(variableName, origin, 
variableData);
                                writeableFile.write(latVar, lat_origin, 
latData);
                                writeableFile.flush();
                        }
                } finally {
                        if (writeableFile != null)
                                writeableFile.close(); // Exception is caught 
here.
                }
#################################

• Follow-Ups:
  • Re: [netcdf-java] ArrayIndexOutOfBoundException when generating a large file
    • From: John Caron