NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Roy Britten wrote:
Hi John, The NetCDF Java (version 2) User's Manual doesn't contain detail of how NetcdfFiles are created and managed over an HTTP connection. Can you point me to any information / documentation? I'm assuming that it's not simply a case of reading the file into memory and working on it from there.
its extemely simple: instead of a RandomAccessFile, the library opens a HTTPRandomAccessFile. this uses the HTTP "Range" command to get ranges of bytes from the remote file. everything else is exactly the same, and no optimizations like caching are done, although i think buffering is done.
see ucar.netcdf.NetcdfFile(URL url) and HTTPRandomAccessFile.java sorry the docs are minimal, an unreleased version has the folowing:"For HTTP access, the URL should begin with http: for example: http://www.unidata.ucar.edu/staff/caron/test/mydata.nc. For HTTP access, the library uses the HTTP "Range" command to get ranges of bytes from the remote file. The efficiency of the remote access depends on how the data is accessed. Reading large contiguous regions of the file should generally be good, while skipping around the file and reading small amounts of data will be poor. In that case you would be better copying the file to a local drive, or putting the file into a DODS server which will more efficiently subset the file on the server, see DODSNetcdfFile, below. For DODS access, the URL should begin with dods: dods://thredds.unidata.ucar.edu/dodsC/test.nc."
The problem I discussed with you last week does not occur when the
NetCDF files are located on another server (Apache/1.3.26 instead of
Apache Tomcat/4.0.1). Unfortunately at the moment we are constrained to
using the Tomcat server.
I note that a small NetCDF file (484 bytes) is acquired with one HEAD
and one GET request, acquisition of a larger file (23100 bytes) fails
after one HEAD and two GET requests with the null pointer exception:
Exception in thread "main" java.lang.NullPointerException
at ucar.netcdf.Attribute.<init>(Attribute.java:119)
at
ucar.netcdf.NetcdfFile.readV1AttributeArray(NetcdfFile.java:771)
at ucar.netcdf.NetcdfFile.readV1VarArray(NetcdfFile.java:1611)
at ucar.netcdf.NetcdfFile.readV1(NetcdfFile.java:1707)
at ucar.netcdf.NetcdfFile.<init>(NetcdfFile.java:188)
at ucar.nc2.NetcdfFile.<init>(NetcdfFile.java:69)
at TestConnection.main(TestConnection.java:18)
it looks like there is an earlier failure that only shows up here. hard to debug without an example i can recreate.
obvious guess is a problem with Tomcat range command, but it may be some minor thing (or an error we arent catching) that is unrelated.
None of the files in our prototype system are (currently) very large. If the HTTPClient library cannot for some reason reliably communicate with a Tomcat server, or vice-versa, an alternative for us would be to create a NetcdfFile from a URLConnection's InputStream. From your knowledge of the NetCDF Java library can you say how complicated you would think development along that path would be?
you cant get random access with an InputStream, in effect you would copy the file and access locally, not very efficient for large files, but ok for small.
netcdf-java archives: