- To: Ethan Davis <edavis@xxxxxxxxxxxxxxxx>
- Subject: Re: [netcdf-java] IOSP open file risk
- From: Doug Lindholm <doug.lindholm@xxxxxxxxxxxxxxxxx>
- Date: Fri, 04 Dec 2009 13:16:51 -0700
I have my own abstract iosp that extends AbstractIOServiceProvider. It
has a single "open" method that works for all my IOSPs. It calls an
"init" method so subclasses have a hook to set up any resources like
database connections. It does nothing with the RandomAccessFile (because
most of my data files have no metadata or aren't files at all). I end up
using the ncml file itself to get the metadata to make the Dimensions
and Variables to add to the NetcdfFile that I get handed in "open". I'd
like to think that NetCDF could define the NetcdfFile for me based
solely on the ncml metadata (not including the "location" data and
whatever metadata it can find there). If I don't like what it gives me,
I can overwrite "open" and clean it up.
On a related note, I would like to have control of the "location" when the ncml specifies a custom iosp. I might want to use a special URI such as a jdbc connection URL. Right now, I set location to "/dev/null" (NetCDF complains if the location doesn't exist) and encode connection info in the iospParam. I'd rather not be limited to data "locations" that can only be represented as a RandomAccessFile.
"Fixes" for these would remove a significant hurdle in an API that otherwise is performing miracles for me. I might be enticed to contribute to the effort.
Thanks, Doug Ethan Davis wrote:
Hi Doug, Doug Lindholm wrote:I don't see an "open" method in AbstractIOServiceProvider. I should have mentioned that I am using netcdf-java 4.Yes, I was suggesting that one be added. EthanThanks, Doug Ethan Davis wrote:Hi Doug, I think the real problem is the lack of documentation on the AbstractIOServiceProvider. The IOServiceProvider is pretty clear that it is the IOSP implementations job to close the RAF in the close() method. But AbstractIOSP doesn't explain what it is doing with the raf variable or the close() method. I thinking adding an open(...) method to AbstractIOServiceProvider that sets the raf variable would both move all the basic raf stuff into AbstractIOServiceProvider and make it easier to understand. Here's the javadoc I just wrote to add to AbstractIOServiceProvider:/** * Abstract base class for IOSP implementations that provides default implementations * of readToByteChannel(...) and readSection(...). * * <p>Implementations should make sure to handle the RandomAccessFile properly by * doing one of the following: * * <ol> * <li> Write your own open(...) and close() methods that keep track of the * RandomAccessFile, be sure to close the RandomAccessFile in your close() * method.</li> * <li> Write your own open(...) and close() methods that call the open(...) * and close() methods defined here, use the "raf" variable also defined * here.</li> * <li> Don't write an open(...) or close() method, so that those defined * here are used.</li> * </ol> * */Any thoughts or suggestions? Thanks, Ethan Doug Lindholm wrote:Hi, Our Tomcat server was barfing with "too many open files." The files (including /dev/null) that were left open where those in my ncml "location." My custom IOSP "close" method simply delegates to super.close(). However, the RandomAccessFile in the superclass that is closed is null. It is not the file that was handed to my IOSP. I see that the "raf" there is protected, so I could set it, but that seems a bit obscure. This seems rather dangerous for the unwitting IOSP developer like me. It seems like you, on the NetCDF side, could set the "raf" in AbstractIOServiceProvider when you construct my IOSP. Otherwise, I would suggest that you make "close" abstract to ensure that it is dealt with. Cheers, Doug
- Follow-Ups:
- Re: [netcdf-java] IOSP open file risk
- From: John Caron
- Re: [netcdf-java] IOSP open file risk
- References:
- [netcdf-java] IOSP open file risk
- From: Doug Lindholm
- Re: [netcdf-java] IOSP open file risk
- From: Ethan Davis
- Re: [netcdf-java] IOSP open file risk
- From: Doug Lindholm
- Re: [netcdf-java] IOSP open file risk
- From: Ethan Davis
- [netcdf-java] IOSP open file risk