NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

Re: [netcdf-java] Variable shape problem & EOFException

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Ok.&nbsp; Could you clarify the version number?&nbsp; I'm on 2.2.22 now - you
mentioned 2.22.14 just now.&nbsp; Is there that much difference between the
two?<br>
<br>
<br>
<br>
John Caron wrote:
<blockquote cite="mid476951BE.60208@xxxxxxxxxxxxxxxx" type="cite">release
2.22.14 should fix this problem. be sure to call flush() after writing
and before reading
  <br>
  <br>
Nick Bower wrote:
  <br>
  <blockquote type="cite">Adding an ncFile.flush() to the submitted
test case as suggested does not resolve the problem.&nbsp; I think it needs
to be marked as a bug until a documented way around it surfaces.
    <br>
    <br>
    <br>
Ethan Davis wrote:
    <br>
&nbsp;
    <blockquote type="cite">Hi Nick,
      <br>
      <br>
I'm not sure if this is a recommended way but you might try to flush()
the NetcdfFileWriteable before reading.
      <br>
      <br>
John might have another answer. He is out of the office this week so
his email may be spotty.
      <br>
      <br>
Hope that helps,
      <br>
      <br>
Ethan
      <br>
      <br>
Nick Bower wrote:
      <br>
&nbsp;&nbsp;&nbsp;
      <blockquote type="cite">Hello.&nbsp; Is the lack of response here
because this is not considered a bug?&nbsp; Is there a another recommended
way perhaps to safely increase a dimension and invalidate the memory
cache in the process?
        <br>
        <br>
Nick
        <br>
        <br>
        <br>
Nick Bower wrote:
        <br>
&nbsp;
        <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
        <blockquote type="cite">I've found a problem in which
cache/memory and disk shape information about variables will disagree
with v2.2.22 of Java Netcdf library.
          <br>
          <br>
When you add a new value to a variable, automatically increasing the
length of a dimension, subsequent reads can throw EOFException because
RandomAccessFile is instructed to read more values than the file
contains - the cached and actual shapes disagree.
          <br>
          <br>
I've created a runnable test case below to explain and demonstrate
success and failure conditions.
          <br>
          <br>
I am getting around this now by not interleaving read/write operations
on variables, but instead reading all variables' data to memory, then
performing any writes I need to after.
          <br>
          <br>
TestInsertRecord.java:
          <br>
          <br>
          <br>
package com.metoceanengineers.datafeeds.netcdf.test;
          <br>
          <br>
import java.io.File;
          <br>
import java.io.IOException;
          <br>
import java.text.DateFormat;
          <br>
import java.text.SimpleDateFormat;
          <br>
          <br>
import junit.framework.TestCase;
          <br>
import ucar.ma2.Array;
          <br>
import ucar.ma2.ArrayInt;
          <br>
import ucar.ma2.DataType;
          <br>
import ucar.nc2.Dimension;
          <br>
import ucar.nc2.NetcdfFileWriteable;
          <br>
          <br>
public class TestInsertRecord extends TestCase {
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DateFormat dateFormat = new 
SimpleDateFormat("yyyyMMdd HHMM");
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; protected NetcdfFileWriteable createNc(String 
prefix) throws
IOException {
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
File mainline = File.createTempFile(prefix+"-", ".nc");
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NetcdfFileWriteable mainlineNc =
NetcdfFileWriteable.createNew(mainline.getAbsolutePath(), false);
          <br>
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Dimension recordsDim =
mainlineNc.addUnlimitedDimension("records");
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
Dimension timeDims[] = {recordsDim};
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Dimension var1Dims[] = {recordsDim}; // 1D
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
mainlineNc.addVariable("time", DataType.INT, timeDims);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; mainlineNc.addVariable("var1", 
DataType.INT, var1Dims);
          <br>
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; mainlineNc.create();
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; return mainlineNc;
          <br>
&nbsp;&nbsp; }
          <br>
          <br>
          <br>
&nbsp;&nbsp; protected String getNcInstance() throws Exception {
          <br>
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NetcdfFileWriteable mainlineNc = 
createNc("testfile");
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
int[] origin = {0};
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
ArrayInt.D1 timeArr = new ArrayInt.D1(2);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; timeArr.set(0, 
(int)dateFormat.parse("20071130
0924").getTime());
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; timeArr.set(1, 
(int)dateFormat.parse("20071130
0926").getTime());
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; mainlineNc.write("time", origin, timeArr);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
ArrayInt.D1 var1Arr = new ArrayInt.D1(2);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; var1Arr.set(0, 10);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; var1Arr.set(1, 12);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; mainlineNc.write("var1", origin, var1Arr);
          <br>
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; mainlineNc.close();
          <br>
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; return mainlineNc.getLocation();
          <br>
&nbsp;&nbsp; }
          <br>
&nbsp;&nbsp;&nbsp;&nbsp; /**
          <br>
&nbsp;&nbsp;&nbsp; * Append new data to end of existing variables.
          <br>
&nbsp;&nbsp;&nbsp; *
          <br>
&nbsp;&nbsp;&nbsp; * @throws Exception
          <br>
&nbsp;&nbsp;&nbsp; */
          <br>
&nbsp;&nbsp; public void testAppendWorksOk() throws Exception {
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
String ncFilename = getNcInstance();
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NetcdfFileWriteable ncFile =
NetcdfFileWriteable.openExisting(ncFilename, false);
          <br>
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /*
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; * Append value (20071130 0924, 11) 
into (time, var1)
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; */
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
ArrayInt.D1 newTimeValue = new ArrayInt.D1(1);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; newTimeValue.set(0, 
(int)dateFormat.parse("20071130
0925").getTime());
          <br>
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ArrayInt.D1 newVarValue = new 
ArrayInt.D1(1);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; newVarValue.set(0, 11);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
int[] origin = {2};
          <br>
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /* The first write will expand the 
variables,
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; * but second write ok as we're just 
writing
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; * and not reading */
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
ncFile.write("time", origin, newTimeValue);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ncFile.write("var1", origin, newVarValue);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
assertEquals(3,
ncFile.findDimension("records").getLength());
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }
          <br>
          <br>
&nbsp;&nbsp; /**
          <br>
&nbsp;&nbsp;&nbsp; * Test insertion of a record in between the 2 existing
          <br>
&nbsp;&nbsp;&nbsp; * records by reading the existing tail, inserting new data
          <br>
&nbsp;&nbsp;&nbsp; * and re-appending.
          <br>
&nbsp;&nbsp;&nbsp; *
          <br>
&nbsp;&nbsp;&nbsp; * Triggers EOFException through interleaved read/writes
          <br>
&nbsp;&nbsp;&nbsp; *
          <br>
&nbsp;&nbsp;&nbsp; * @throws Exception
          <br>
&nbsp;&nbsp;&nbsp; */
          <br>
&nbsp;&nbsp; public void testInsertFails() throws Exception {
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
String ncFilename = getNcInstance();
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NetcdfFileWriteable ncFile =
NetcdfFileWriteable.openExisting(ncFilename, false);
          <br>
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ArrayInt.D1 newTimeValue = new 
ArrayInt.D1(1);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; newTimeValue.set(0, 
(int)dateFormat.parse("20071130
0925").getTime());
          <br>
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ArrayInt.D1 newVarValue = new 
ArrayInt.D1(1);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; newVarValue.set(0, 11);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
/* Going to insert at 1, so read existing value,
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; * write down new one, and re-append 
old tail.
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; */
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
int[] insertPointOrigin = {1};
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; int[] appendOrigin = {2};
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; int[] shape = {1};
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
Array tailTime =
ncFile.findVariable("time").read(insertPointOrigin, shape);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ncFile.write("time", insertPointOrigin, 
newTimeValue);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ncFile.write("time", appendOrigin, 
tailTime);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
/* Next line excepts - why?&nbsp; Because the last write above
at
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; * records index 2 triggers an 
increase in the CACHED/MEMORY
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; * length of all variables to 3, but 
on disk it's still the
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; * original length 2.
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; *
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; *&nbsp; Therefore we get 
EOFException.
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; */
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
Array tailVar1 =
ncFile.findVariable("var1").read(insertPointOrigin, shape);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ncFile.write("var1", insertPointOrigin, 
newVarValue);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ncFile.write("var1", appendOrigin, 
tailVar1);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
assertEquals(3,
ncFile.findDimension("records").getLength());
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }
          <br>
          <br>
}
          <br>
          <br>
          <br>
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </blockquote>
_______________________________________________
        <br>
netcdf-java mailing list
        <br>
<a class="moz-txt-link-abbreviated" 
href="mailto:netcdf-java@xxxxxxxxxxxxxxxx";>netcdf-java@xxxxxxxxxxxxxxxx</a>
        <br>
For list information or to unsubscribe, visit:
<a class="moz-txt-link-freetext" 
href="http://www.unidata.ucar.edu/mailing_lists/";>http://www.unidata.ucar.edu/mailing_lists/</a>&nbsp;&nbsp;
 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </blockquote>
    </blockquote>
    <br>
    <br>
_______________________________________________
    <br>
netcdf-java mailing list
    <br>
<a class="moz-txt-link-abbreviated" 
href="mailto:netcdf-java@xxxxxxxxxxxxxxxx";>netcdf-java@xxxxxxxxxxxxxxxx</a>
    <br>
For list information or to unsubscribe, visit:
<a class="moz-txt-link-freetext" 
href="http://www.unidata.ucar.edu/mailing_lists/";>http://www.unidata.ucar.edu/mailing_lists/</a>
 &nbsp; </blockquote>
</blockquote>
<br>
<br>
<div class="moz-signature">-- <br>
<p><span style="font-size: 10pt; font-family: Verdana,sans-serif;">Regards,
Nick</span><br>
<span style="color: rgb(204, 204, 
204);">____________________________________</span></p>
<p><span style="font-size: 10pt; font-family: Verdana,sans-serif;">Dr.
Nicholas Bower</span>
<span style="font-size: 8pt; font-family: Verdana,sans-serif;">Ph.D.<br>
<span
 style="font-size: 8pt; font-family: Verdana,sans-serif; font-style: 
italic;">Senior
Software Developer/Integrator, IT</span><br>
<br>
<a 
href="mailto:nick.bower@xxxxxxxxxxxxxxxxxxxxx";>nick.bower@xxxxxxxxxxxxxxxxxxxxx</a></span></p>
<img style="border: 1px solid black;"
 src="cid:part1.07080401.04090103@metoceanengineers.com" alt="MOELogo"><br>
<p><span style="font-size: 10pt; font-family: Verdana,sans-serif;"><b>RPS
Metocean Engineers</b><br>
31 Bishop Street, Jolimont WA 6014<br>
<b>Phone:</b> +61 8 9387 7955 &nbsp;&nbsp;&nbsp;<b>Fax:</b> +61 8 9387 
6686</span></p>
</div>
</body>
</html>

JPEG image

  • 2007 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: