Re: [netcdf-java] no IOException for shortened file

• To: Bob Simons - NOAA Federal <bob.simons@xxxxxxxx>
• Subject: Re: [netcdf-java] no IOException for shortened file
• From: Christian Ward-Garrison <cwardgar@xxxxxxxx>
• Date: Wed, 2 Nov 2016 01:16:06 -0600

Hi Bob,

Sorry for the delay in responding. I would have expected an exception as
well, but apparently this is working as intended. Specifically, we're
trying to match the behavior of the netcdf C library, which allows
truncated files [1]. I verified this with ncdump, which printed all data
values without error. For truncated values, ncdump printed '0', which is
the default for floats in C. NetCDF-Java, on the other hand, used whatever
values were left in the RandomAccessFile buffer after the last successful
read for truncated values. Maybe that's a problem.

If you like, it's possible to disallow truncated files by setting
ucar.nc2.iosp.netcdf3.N3header.disallowFileTruncation to true, in which
case you'll get the exception you're expecting.

Cheers,
Christian

[1]
http://www.unidata.ucar.edu/software/netcdf/netcdf-4/newdocs/netcdf-c/Fill-Values.html#Fill-Values

On Tue, Oct 18, 2016 at 11:56 AM, Bob Simons - NOAA Federal <
bob.simons@xxxxxxxx> wrote:

> In our system, and I suspect elsewhere, sometimes .nc files don't get
> completely copied from one place to another. Normally, this is caught and
> dealt with (e.g., by the script doing the copying), but sometimes it isn't
> (e.g., a computer is restarted while a script is running), leaving a
> shortened file in place.
>
> I wanted to understand how netcdf-java (v4.6.6) reacted to these shorted
> files. So I made the attached file by copying only the first 1/6th of a
> valid file. Then I tried to read it with this code
>
>             NetcdfFile ncFile = NetcdfFile.open("/git/erddapTest/nc/
> invalidShortened.nc");
>             try {
>                 Variable var = ncFile.findVariable("MWcdom");
>  //size=[1,1,2321,4001]
>                 Array array = var.read();
>                 System.out.println("shape=" + Arrays.toString(array.
> getShape()));
>
>                 array = var.read("0,0,0:2000:1000,0:4000:1000");
>                 System.out.println("shape=" + Arrays.toString(array.
> getShape()));
>
>                 //System.out.println(array.toString());
>                 System.out.println("Shouldn't get here!");
>                 ncFile.close();
>             } catch (Throwable t) {
>                 try {
>                     ncFile.close();
>                 } catch (Throwable t2) {
>                 }
>                 System.out.println("caught: " + t.toString());
>             }
>
> Note that Variable.read() documentation says that the method throws an
> IOException.
>
> To my surprise, the above code doesn't throw any exception.
> netcdf-java says it can read the entire var and read the specified section
> (which gets values from throughout the variable).
>
> Is that what you expected?
> Am I doing something wrong in my test?
> Doesn't netcdf-java check the validity of the file?
> Why don't the read()s cause an IOException?
>
> Thank you for looking into this.
>
> --
> Sincerely,
>
> Bob Simons
> IT Specialist
> Environmental Research Division
> NOAA Southwest Fisheries Science Center
> 99 Pacific St., Suite 255A      (New!)
> Monterey, CA 93940               (New!)
> Phone: (831)333-9878            (New!)
> Fax:   (831)648-8440
> Email: bob.simons@xxxxxxxx
>
> The contents of this message are mine personally and
> do not necessarily reflect any position of the
> Government or the National Oceanic and Atmospheric Administration.
> <>< <>< <>< <>< <>< <>< <>< <>< <><
>
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web.  Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> netcdf-java mailing list
> netcdf-java@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>

• Follow-Ups:
  • Re: [netcdf-java] no IOException for shortened file
    • From: Bob Simons - NOAA Federal

• References:
  • [netcdf-java] no IOException for shortened file
    • From: Bob Simons - NOAA Federal