NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

Re: [netcdf-java] IOSP open file risk

I have my own abstract iosp that extends AbstractIOServiceProvider. It has a single "open" method that works for all my IOSPs. It calls an "init" method so subclasses have a hook to set up any resources like database connections. It does nothing with the RandomAccessFile (because most of my data files have no metadata or aren't files at all). I end up using the ncml file itself to get the metadata to make the Dimensions and Variables to add to the NetcdfFile that I get handed in "open". I'd like to think that NetCDF could define the NetcdfFile for me based solely on the ncml metadata (not including the "location" data and whatever metadata it can find there). If I don't like what it gives me, I can overwrite "open" and clean it up.
On a related note, I would like to have control of the "location" when the ncml specifies a custom iosp. I might want to use a special URI such as a jdbc connection URL. Right now, I set location to "/dev/null" (NetCDF complains if the location doesn't exist) and encode connection info in the iospParam. I'd rather not be limited to data "locations" that can only be represented as a RandomAccessFile.
"Fixes" for these would remove a significant hurdle in an API that otherwise is performing miracles for me. I might be enticed to contribute to the effort. 

Thanks,
Doug

Ethan Davis wrote:

Hi Doug,

Doug Lindholm wrote:

I don't see an "open" method in AbstractIOServiceProvider. I should have
mentioned that I am using netcdf-java 4.


Yes, I was suggesting that one be added.

Ethan


Thanks,
Doug

Ethan Davis wrote:

Hi Doug,

I think the real problem is the lack of documentation on the
AbstractIOServiceProvider. The IOServiceProvider is pretty clear that it
is the IOSP implementations job to close the RAF in the close() method.
But AbstractIOSP doesn't explain what it is doing with the raf variable
or the close() method.

I thinking adding an open(...) method to AbstractIOServiceProvider that
sets the raf variable would both move all the basic raf stuff into
AbstractIOServiceProvider and make it easier to understand.

Here's the javadoc I just wrote to add to AbstractIOServiceProvider:


/**
 * Abstract base class for IOSP implementations that provides default
implementations
 * of readToByteChannel(...) and readSection(...).
 *
 * <p>Implementations should make sure to handle the RandomAccessFile
properly by
 * doing one of the following:
 *
 * <ol>
 *   <li> Write your own open(...) and close() methods that keep
track of the
 *     RandomAccessFile, be sure to close the RandomAccessFile in
your close()
 *     method.</li>
 *   <li> Write your own open(...) and close() methods that call the
open(...)
 *     and close() methods defined here, use the "raf" variable also
defined
 *     here.</li>
 *   <li> Don't write an open(...) or close() method, so that those
defined
 *     here are used.</li>
 * </ol>
 *
 */

Any thoughts or suggestions?

Thanks,

Ethan

Doug Lindholm wrote:

Hi,

Our Tomcat server was barfing with "too many open files." The files
(including /dev/null) that were left open where those in my ncml
"location." My custom IOSP "close" method simply delegates to
super.close(). However, the RandomAccessFile in the superclass that is
closed is null. It is not the file that was handed to my IOSP. I see
that the "raf" there is protected, so I could set it, but that seems a
bit obscure.

This seems rather dangerous for the unwitting IOSP developer like me. It
seems like you, on the NetCDF side, could set the "raf" in
AbstractIOServiceProvider when you construct my IOSP. Otherwise, I would
suggest that you make "close" abstract to ensure that it is dealt with.

Cheers,
Doug