Re: In-memory reading of NetCDF files

• To: "Janke" <janke@xxxxxxxxxxx>, <netcdf-java@xxxxxxxxxxxxxxxx>
• Subject: Re: In-memory reading of NetCDF files
• From: "John Caron" <caron@xxxxxxxxxxxxxxxx>
• Date: Mon, 5 Nov 2001 13:05:03 -0700

Hi Andrew:

We can read netcdf files remotely using the HTTP "range" command. While this
probably wont satisfy your efficiency needs, look at
ucar.netcdf.HTTPRandomAccessFile (below); i think you can just implement the
equivalent. Its surprisingly small.  Also see the ucar.netcdf.NetcdfFile(URL
url) constructor.

I think it would be a very cool addition, we'd probably be happy to
integrate it into the core library.

BTW, the ucar.nc2 package is a newer API that is built on top of
ucar.netcdf. One advantage is that it uses ucar.ma2 which is a lot more
efficient than ucar.multiarry in some cases. See
http://www.unidata.ucar.edu/packages/netcdf-java/ for details.


----- Original Message -----
From: "Janke" <janke@xxxxxxxxxxx>
To: <netcdf-java@xxxxxxxxxxxxxxxx>
Sent: Monday, November 05, 2001 11:55 AM
Subject: In-memory reading of NetCDF files


> NetCDF developers,
>
>   I am a Java developer writing an application that reads
> NetCDF files as one of its data sources. In some cases,
> these NetCDF files are downloaded from a network source
> before being read in. In our current implementation, the
> NetCDF files are saved to temporary files on disk, and then
> read in using a call to the NetcdfFile(File file, boolean readonly)
> constructor. For efficiency purposes, I'd like to skip the
> saving to tempfile step, and read in the Netcdf from a byte[]
> in memory (which the application has as soon as the file
> is downloaded over the network).
>
>   Is there a way to do this with the current NetCDF Java
> packages? As far as I can tell, there's no way to make a
> java.io.File refer to in-memory data, as it's an abstraction
> of pathnames. If this functionality isn't available, is it something
> the community would find useful? If we implemented this
> as an extension to the current NetCDF packages, we would
> release it under the LGPL as required.
>
> Andrew Janke
> Computational Geosciences, Inc.
>

// $Id: HTTPRandomAccessFile.java,v 1.2 2001/05/17 15:15:08 steve Exp $
/*
 * Copyright 1997-2000 Unidata Program Center/University Corporation for
 * Atmospheric Research, P.O. Box 3000, Boulder, CO 80307,
 * support@xxxxxxxxxxxxxxxx.
 *
 * This library is free software; you can redistribute it and/or modify it
 * under the terms of the GNU Lesser General Public License as published by
 * the Free Software Foundation; either version 2.1 of the License, or (at
 * your option) any later version.
 *
 * This library is distributed in the hope that it will be useful, but
 * WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU Lesser
 * General Public License for more details.
 *
 * You should have received a copy of the GNU Lesser General Public License
 * along with this library; if not, write to the Free Software Foundation,
 * Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
 */

/*
 * HTTPRandomAccessFile.java.
 * @author John Caron, based on work by Donald Denbo
 */

package ucar.netcdf;

import HTTPClient.*;

import java.io.FileNotFoundException;
import java.io.FileDescriptor;
import java.io.IOException;
import java.net.URL;

/**
 * HTTPRandomAccessFile.java.
 * @author John Caron, based on work by Donald Denbo
 */

public class HTTPRandomAccessFile extends RandomAccessFile {

  private long total_length = 0;
  private HTTPConnection conn;
  private String path;
  private NVPair[] form = null;
  private NVPair[] header = new NVPair[2];

  public HTTPRandomAccessFile(URL url) throws IOException {
    this( url, defaultBufferSize);
  }

  public HTTPRandomAccessFile(URL url, int bufferSize) throws IOException {
    super( bufferSize);
    file = null;

    path = url.getFile(); // not "getPath()" to accomodate JDK 1.2
    conn = new HTTPConnection(url);
    try {
      HTTPResponse test = conn.Head(path);
      if(test.getStatusCode() == 404)
        throw new FileNotFoundException(test.getReasonLine());
      if(test.getStatusCode() >= 300) {
        throw new IOException(test.getReasonLine());
      }
      total_length = test.getHeaderAsInt("Content-Length");
    } catch (ModuleException me) {
      me.printStackTrace();
      throw new IOException();
    }
    header[0] = new NVPair("User-Agent", "HTTPnetCDF;");
  }

  protected int read_(long pos, byte[] buff, int off, int len) throws
IOException {
    long end = pos + len - 1;
    if (end >= total_length)
      end = total_length - 1;

    byte[] data = null;
    header[1] = new NVPair("Range", "bytes="+pos+"-"+end);
    //System.out.print(" want = "+pos+"-"+end+": ");

    try {
      HTTPResponse res = conn.Get(path, form, header);
      if(res.getStatusCode() >= 300) {
        System.out.println(new String(res.getData()));
        throw new IOException(res.getReasonLine());
      }
      data = res.getData();
      //System.out.println(res.getHeader("Content-Range"));
    } catch (ModuleException me) {
      me.printStackTrace();
      throw new IOException(me.getMessage());
    }

    // copy to output buffer
    int reslen = Math.min( len, data.length);
    System.arraycopy( data, 0, buff, off, reslen );

    return reslen;
  }

  public long length( ) throws IOException {
    long fileLength = total_length;
    if( fileLength < dataEnd )
      return dataEnd;
    else
      return fileLength;
  }


  /**
   * override the rest of the RandomAccessFile public methods
   */
  public void close() {
  }

  public FileDescriptor getFD() {
    return null;
  }

  /**
   * implement HTTP access.
   */

}