Re: [netcdf-java] ArrayIndexOutOfBoundException when generating a large file

• To: Jean-Michel.Perraud@xxxxxxxx, netcdf-java@xxxxxxxxxxxxxxxx
• Subject: Re: [netcdf-java] ArrayIndexOutOfBoundException when generating a large file
• From: John Caron <caron@xxxxxxxxxxxxxxxx>
• Date: Sat, 02 May 2009 14:50:27 -0600

Hi Jean-Michel:

ok, finally found it - an overflow of an int variable in some obscure place. i 
will get a new release out on monday that will fix.

BTW, you can cut your time approx in half by using:

   NetcdfFileWriteable.setFill(false);


thanks for debugging!

Jean-Michel.Perraud@xxxxxxxx wrote:

Hi John,

I was not using setLargeFile(true) indeed. I still get the same exception with 
it added. I attach the test program I use. I ran it both from a WinXP 32 bit 
workstation and a Windows Server 2003 R2 x64 with the same behavior. The JVM is:
java version "1.6.0_11"
Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
Java HotSpot(TM) Client VM (build 11.0-b16, mixed mode, sharing)

Thanks for your help.

Cheers,
Jean-Michel
-----Original Message-----

From: John Caron [mailto:caron@xxxxxxxxxxxxxxxx] Sent: Thursday, 30 April 2009 9:37 PM

To: Perraud, Jean-Michel (CLW, Black Mountain)
Cc: netcdf-java@xxxxxxxxxxxxxxxx
Subject: Re: [netcdf-java] ArrayIndexOutOfBoundException when generating a 
large file

Hi Jean-Michel:

Are you setting NetcdfFileWriteable.setLargeFile(true) ?
Can i see your createLatLonTimeDataCube() method?

Jean-Michel.Perraud@xxxxxxxx wrote:

Hi,

I want to generate a lat*lon*time data file which I estimate will be ~16GB. I 
am using the latitude as an unlimited dimension to (1) lay out the file for 
efficient reading of time series and (2) as I understand, overcome the 4GB size 
limit. Things work fine with small data. Testing with lat=300, lon=300 and 
time=40000 the file fails to grow beyond 8GB. This is only my second 
significant use of netcdf and I am probably missing something obvious to 
seasoned netCDF users.

I am using netcdfAll-4.0.jar. The stack trace in Eclipse is from the close() 
method, but things must go pear-shaped before as only half the records are 
output (179 out of 300 latitudes).

Any advice would be appreciated. Sample code/netcdf structure follows.
Cheers
-------------------------------------------
Jean-Michel Perraud
Prediction and Reporting Technologies, CSIRO Land and Water


Thread [main] (Suspended (exception ArrayIndexOutOfBoundsException))    
        RandomAccessFile.write(int) line: 747    // 
ArrayIndexOutOfBoundsException: -2013020
        RandomAccessFile.writeInt(int) line: 1438       
        N3header.writeNumrecs() line: 900       
        N3raf(N3iosp).flush() line: 869 
        NetcdfFileWriteable.flush() line: 731   
        NetcdfFileWriteable.close() line: 741   
        TestLargeGeneration.main(String[]) line: 55     

netcdf file://XXXX/tmp/test.nc {
 dimensions:
   time = 40000;
   lat = UNLIMITED;   // (179 currently)
   lon = 300;
 variables:
   float testVar(lat=179, lon=300, time=40000);
     :long_name = "testVar";
     :units = "mm";
     :missing_value = -9999.0f; // float
     :_FillValue = -9999.0f; // float
   char cellId(lat=179, lon=300);
     :long_name = "Cell ID";
   float lat(lat=179);
     :units = "degrees_north";
     :axis = "Y";
     :standard_name = "latitude";
     :_CoordinateAxisType = "Lat";
   float lon(lon=300);
     :units = "degrees_east";
     :axis = "X";
     :standard_name = "longitude";
     :_CoordinateAxisType = "Lon";
   int time(time=40000);
     :units = "days since 1889-01-01";
     :axis = "T";
     :standard_name = "time";
     :long_name = "time";
     :_CoordinateAxisType = "Time";
}
#################################
                try {
                        writeableFile = createLatLonTimeDataCube(filename);
                        // The outer dimension has shape 1, since we will write 
one record
                        // at a time, and one
                        // record corresponds to one latitude ordinate
                        ArrayFloat.D3 variableData = new ArrayFloat.D3(1, 
LON_LEN, TIME_LEN);
                        // ArrayObject.D2 cellIdData = new
                        // ArrayObject.D2(class(java.lang.String), 
latDim.getLength(),
                        // lonDim.getLength());
                        // An array to record the latitude ordinate
                        ucar.ma2.Array latData = 
ucar.ma2.Array.factory(DataType.FLOAT,
                                        new int[] { 1 });

                        // The origin to use to write the runoff record
                        int[] origin = new int[] { 0, 0, 0 }; // lat, lon, time
                        // The origin to use to write the latitude for each 
record
                        int[] lat_origin = new int[] { 0 };

                        for (int lat = 0; lat < LAT_LEN; lat++) {
                                latData.setFloat(latData.getIndex(), (float) 
(lat / 10.0));
                                for (int lon = 0; lon < LON_LEN; lon++) {
                                        for (int time = 0; time < TIME_LEN; 
time++) {
                                                variableData.set(0, lon, time, 
(float) time);
                                        }
                                }
                                // write the data out for this record
                                origin[0] = lat;
                                lat_origin[0] = lat;
                                writeableFile.write(variableName, origin, 
variableData);
                                writeableFile.write(latVar, lat_origin, 
latData);
                                writeableFile.flush();
                        }
                } finally {
                        if (writeableFile != null)
                                writeableFile.close(); // Exception is caught 
here.
                }
#################################


_______________________________________________
netcdf-java mailing list
netcdf-java@xxxxxxxxxxxxxxxx

For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/

• References:
  • [netcdf-java] ArrayIndexOutOfBoundException when generating a large file
    • From: Jean-Michel.Perraud
  • Re: [netcdf-java] ArrayIndexOutOfBoundException when generating a large file
    • From: John Caron