NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

[netcdf-java] Improper writing of bytes to unlimited variables in 4.2.18

I've encountered a problem writing values of type byte to a variable with a single unlimited dimension to a netcdf-3 format file, in revision 4.2.18 (and probably earlier versions as well). It appears to improperly pad single-byte values out to 4-byte words. If I don't make the dimension unlimited, I don't have this problem. 

Writing a simple array of consecutive byte values gives me this:

netcdf test-byte-unlimited {
dimensions:
        X = 5 ;
        D = UNLIMITED ; // (18 currently)
variables:
        double X(X) ;
        byte V(D) ;
data:

 X = _, _, _, _, _ ;

 V = 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0, 4, 0, 0, 0, 5, 0 ;
}

I'm expecting:
 V = 1, 2, 3, 4, 5, 6, 7, 8, 9, -1, -2, -3, -4, -5, -6, -7, -8, -9 ;
Following is a patch that appears to solve the problem for me, followed by my test-case code. It seems this issue has come up; there was a commented-out fix for padding of byte values, but it only applied for files with a single variable; I really need to have this working for files with other fixed-dimension variables in them as well.
Note that the test case below does indeed pass; re-reading the written variable using NetCDF-Java returns the entire array without the padding. With the padding, though, the file size is larger (186 vs 240 bytes on my system), and the C API shows incorrect values.
This is the first time I've looked at NetCDF library code of any kind - it will probably be obvious that I'm not familiar with the file format internals. 

Here's the patch:
diff -rupN ncSrc-4.2.18-base/cdm/src/main/java/ucar/nc2/iosp/netcdf3/N3header.java ncSrc-4.2.18/cdm/src/main/java/ucar/nc2/iosp/netcdf3/N3header.java --- ncSrc-4.2.18-base/cdm/src/main/java/ucar/nc2/iosp/netcdf3/N3header.java 2010-12-17 10:08:16.000000000 -0800 +++ ncSrc-4.2.18/cdm/src/main/java/ucar/nc2/iosp/netcdf3/N3header.java 2011-01-14 16:54:17.000000000 -0800 
@@ -883,18 +883,10 @@ public class N3header {
       raf.writeInt(n);
     }
- // Note on padding: In the special case of only a single record variable of character, byte, or short 
-    // type, no padding is used between data values.
-    boolean usePadding = true;
-    /* if (n == 1) {
-      Variable var = vars.get(0);
-      DataType dtype = var.getDataType();
- if ((dtype == DataType.CHAR) || (dtype == DataType.BYTE) || (dtype == DataType.SHORT)) 
-        usePadding = false;
-    } */
-
     for (int i = 0; i < n; i++) {
       Variable var = vars.get(i);
+      DataType dtype = var.getDataType();
+
       writeString(var.getName());

       // dimensions
@@ -908,7 +900,8 @@ public class N3header {
         if (!dim.isUnlimited())
           vsize *= dim.getLength();
       }
-      if (usePadding)
+      if (!var.isUnlimited() ||
+ ((dtype != DataType.CHAR) && (dtype != DataType.BYTE) && (dtype != DataType.SHORT))) 
         vsize += padding(vsize);

       // variable attributes



------
Test case:

    @Test
    public void byteReadWriteUnlimitedDim() throws Exception {
        File f = new File("/tmp/test-byte-unlimited.nc");
byte[] data = {1, 2, 3, 4, 5, 6, 7, 8, 9, -1, -2, -3, -4, -5, -6, -7, -8, -9}; 
        if (f.exists()) {
            f.delete();
        }
NetcdfFileWriteable outf = NetcdfFileWriteable.createNew(f.getPath()); 
        //Dimension d = outf.addDimension("D", data.length);
        Dimension d0 = outf.addDimension("X", 5);
        Dimension d = outf.addUnlimitedDimension("D");
Variable v0 = outf.addVariable("X", DataType.DOUBLE, new Dimension[]{d0}); Variable v = outf.addVariable("V", DataType.BYTE, new Dimension[]{d}); 
        assertEquals(1, v.getElementSize());
        outf.create();
Array arr = Array.factory(DataType.BYTE, new int[]{data.length}, data); 
        outf.write(v.getName(), arr);
        outf.close();

        NetcdfFile ncf = NetcdfFile.open(f.getPath());
        Variable inv = ncf.findVariable("V");
        assertEquals(inv.getDataType(), DataType.BYTE);
        int[] org = {0};
byte[] readdata = (byte[]) inv.read(org, inv.getShape()).copyTo1DJavaArray(); 

        assertEquals(1, inv.getElementSize());
        assertArrayEquals(data, readdata);
        // this test passes, but ncdump shows improper zero-padding
    }


--------



--
Chris Chamberlin  <chris.chamberlin@xxxxxxxx>
NOAA Center for Tsunami Research, NOAA PMEL/UW JISAO
7600 Sand Point Way NE Bldg 3, Seattle, WA 98115 USA
+1 206-526-6809          http://nctr.pmel.noaa.gov/