RE: 19990929: Write/Read of 2D pages

• To: "Lyn Greenhill" <lyng@xxxxxxxxxxxxxxxx>
• Subject: RE: 19990929: Write/Read of 2D pages
• From: "John Caron" <caron@xxxxxxxx>
• Date: Fri, 1 Oct 1999 14:25:46 -0600

Hi Lyn, i just got back from vacation.

> Hello:
>
> Below is the text of a message I sent about a week ago to
> John Caron, and I
> have yet to receive a reply.  I'm assuming he's busy.  If
> anyone could help
> me, I would really appreciate the advice.
>
> To put this into context, I asked John several months ago
> about the very
> slow performance of our I/O with netCDF.  He replied that it
> shouldn't be
> slow unless we have programmed something dumb.  I can send my original
> query and his reply if you are interested.  He offered to
> reply to future
> queries regarding this problem, and below is my latest posting.
>
>
> I have spent some time delving into the netCDF I/O that we
> are doing, and I
> believe I have found some serious time wasters in our code.
> I will show
> you below what I mean.  Anyway, I am somewhat stuck on retrieving
> information from a netCDF file, and I am hoping you might
> point me in the
> right direction.
>
> The poor performance in our program, as I mentioned earlier,
> is do to the
> use of nested loops to write/read multidimensional data.  I
> have been going
> through the code and replacing this logic with equivalent
> MultiArray calls
> to write the data in one pass.  For example, the original
> code to write a
> 3-dimensional array looked like this:
>
> private void saveTripleArrayDoubleData(NetcdfFile cdfFile,
> String dataName,
> double[][][] data)     {
>     try {
>         Variable storage = cdfFile.get(dataName);
>         if (storage == null) throw new IOException();
>         int[] numberIndex = new int[3];
>         int totalX = data.length;
>         int totalY = data[0].length;
>         int totalZ = data[0][0].length;
>
>         for (int x = 0; x < totalX; x++) {
>             numberIndex[0] = x;
>             for (int y = 0; y < totalY; y++) {
>                 numberIndex[1] = y;
>                 for (int z = 0; z < totalZ; z++) {
>                     numberIndex[2] = z;
>                     storage.setDouble(numberIndex, data[x][y][z]);
>                 }
>             }
>         }
>     } catch(java.io.IOException exception) {}
>     catch(java.lang.NullPointerException exception) {}
> }
>
> As you can see in this method, each element of the 3D array is set
> individually, which as you can imagine, takes forever, because this
> operation can't be buffered.  The original code used similar
> logic to read
> data back in, using nested loops and the getDouble() element accessor.
>
> I have replaced this code with the following method:
>
> private void saveTripleArrayDoubleData(NetcdfFile cdfFile,
> String dataName,
> double[][][] data) {
>     try {
>         Variable storage = cdfFile.get(dataName);
>         if (storage == null) throw new IOException();
>         // use multiarray to write data all at once, rather
> than one at a time
>         MultiArray dataMa = new ArrayMultiArray(data);
>         int[] origin = {0, 0, 0};
>         storage.copyin(origin, dataMa);
>
>     } catch(java.io.IOException exception) {}
>     catch(java.lang.NullPointerException exception) {}
> }
>
> I have tested this code and it works great.  The problem I'm
> having now is
> with accessing the data I've written.

this looks good to me

>
> I'm not a seasoned veteran using Java, so please bear with
> me.  To read
> back the MultiArray data, the method copyout() is used.  There are two
> parameters, the origin and shape, both int arrays.  I think I
> understand
> what both of these are doing, the first passing where to
> start reading, and
> the second the actual dimensions of the MultiArray.  So far,
> no problem.
>
> My problem involves converting the returned MultiArray into
> something I can
> use in my program.  When we write the 3D array, what we are doing is
> writing "pages" of 2D arrays, and when we want to read in the
> data, we only
> want to access a single page.  The first dimension controls
> which page.  In
> the example code you have on the web site, the temperature
> array (double
> T(time, lat, lon)), is similar in structure to what we're
> doing, in that
> each time is a page, and we want to access a particular time
> and read back
> in the 2D array corresponding the (lat, lon) temperatures.
> Right now, I'm
> setting the first value in the origin array to the particular
> page I want
> to retrieve (which matches the organization in the schema).  I haven't
> tested this yet to make sure it is doing what I want.

so you set  origin = {pageno, 0, 0)
                shape =  {1, nx, ny}

and you get a 2D MultiArray of dimension nx by ny.
At this point, the data is memory resident, and you could access the
data using
        data.getDouble(index);
which would be my recommendation, since it avoids further data copying.

However if you need to pass a double array to some other routine, you
can use toArray() to get a Java array, but it is a 1D, not 2D array. I
think in fact that it should construct the 2D array, but currently it
doesnt.

So you would unfortunately have to transfer this into a 2D array
yourself, in which case you might as well loop over the
data.getDouble(index) call rather than use toArray().