Re: 20050329:Is it possible to archive streaming data using NetCDF?

• To: Peyush Jain <peyush.jain@xxxxxxxx>
• Subject: Re: 20050329:Is it possible to archive streaming data using NetCDF?
• From: John Caron <caron@xxxxxxxxxxxxxxxx>
• Date: Tue, 29 Mar 2005 12:49:19 -0700

Unidata Support wrote:

------- Forwarded Message

To: support-netcdf-java@xxxxxxxxxxxxxxxx
From: "Peyush Jain" <peyush.jain@xxxxxxxx>
Subject: netCDF Java - Newbie Question: Is it possible to archive streaming 
data using NetCDF?
Organization: UCAR/Unidata
Keywords: 200503291623.j2TGNUiv002216

Institution: NASA
Package Version: Version 2.1
Operating System: Windows XP Pro
Hardware Information: P4, 3.0GHz, 2GB RAM
Inquiry: Hello,

I was wondering if it is possible to archive streaming data using netCDF. If it 
is, can you please point me to an example? A

example below

ssuming the data coming in has 5 one dimensional arrays (of different types), I 
was able to create 5 variables and save them to a netCDF file. Everytime I 
receive this data, it overwrites the previous data in the file. I am sure that 
it is not appending the data because the file size never increases.

I am using the following function to write to netCDF file:
ncfile.write(name, ArrayAbstract.factory(doubleArray));

problem is you are using this call, where origin assumed to be 0

  * @param values write this array; must be same type and rank as Variable
  * @throws IOException
  */

 /** Write data to the named variable. Must not be in define mode.

  * @param origin offset within the variable to start writing.
  * @param values write this array; must be same type and rank as Variable
  * @throws IOException
  */

 public void testNC3WriteWithRecord() throws IOException {

   // define dimensions, including unlimited
   Dimension latDim = ncfile.addDimension("lat", 64);
   Dimension lonDim = ncfile.addDimension("lon", 128);
   Dimension timeDim = ncfile.addDimension("time", -1);

   // define Variables
   Dimension[] dim3 = new Dimension[3];
   dim3[0] = timeDim;
   dim3[1] = latDim;
   dim3[2] = lonDim;

   // double T(time, lat, lon) ;
   //   T:long_name="surface temperature" ;
   //   T:units = "degC" ;
   ncfile.addVariable("T", DataType.DOUBLE, dim3);
   ncfile.addVariableAttribute("T", "long_name", "surface temperature");
   ncfile.addVariableAttribute("T", "units", "degC");


   // float lat(lat) ;
   //   lat:units = "degrees_north" ;
   ncfile.addVariable("lat", DataType.FLOAT, new Dimension[] {latDim});
   ncfile.addVariableAttribute("lat", "units", "degrees_north");

   // float lon(lon) ;
   // lon:units = "degrees_east" ;
   ncfile.addVariable("lon", DataType.FLOAT, new Dimension[] {lonDim});
   ncfile.addVariableAttribute("lon", "units", "degrees_east");

   // int time(time) ;
   //   time:units = "hours" ;
   ncfile.addVariable("time", DataType.INT, new Dimension[] {timeDim});
   ncfile.addVariableAttribute("time", "units", "hours");

   //  :title = "Example Data" ;
   ncfile.addGlobalAttribute("title", "Example Data");

   // create the file
   try {
     ncfile.create();
   }  catch (IOException e) {
     System.err.println("ERROR creating file");
     e.printStackTrace();
     return;
   }
   System.out.println( "ncfile = "+ ncfile);

   // now write one record at a time
   Variable v = ncfile.findTopVariable("T");

   ArrayInt timeData = new ArrayInt.D1(1);
   int[] origin = new int[v.getRank()];
   int[] timeOrigin = new int[1];

   for (int time=0; time<100; time++) {
     // fill the data array
     Index ima = data.getIndex();
     for (int j=0; j<latDim.getLength(); j++)
       for (int k=0; k<lonDim.getLength(); k++)
         data.setDouble(ima.set(0,j,k), (double) time*j*k);
     timeData.setInt( timeData.getIndex(), time);

     // write to file
     origin[0] = time;
     timeOrigin[0] = time;
     try {
       ncfile.write("T", origin, data);
       ncfile.write("time", timeOrigin, timeData);
     } catch (IOException e) {
       System.err.println("ERROR writing file");
     } catch (InvalidRangeException e) {
       e.printStackTrace();
     }
   }

   // all done
   try {
     ncfile.close();
   } catch (IOException e) {
     System.err.println("ERROR writing file");
   }

   System.out.println( "**** TestWriteRecord done");
 }