NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Jon Blower wrote:
Thanks John - just to clarify, at the moment files are only removed if the number of elements is exceeded, and their age in the cache is disregarded?
correct
1. The IOSP builds the entire NetcdfFile object each time its opened, rather than building only on demand. Netcdf-3 files are quite fast, but other IOSPs can be much slower. Recently weve been considering a "fast open" option, but it may be hard to implement.Out of interest, what was the main motivation for creating the cache system, i.e. what is the expensive operation that you are optimizing?
2. The OS file handle must be opened each time.Caching apparently reduces average latency by about 30% (perhaps more). Not sure about throughput or server resource usage.
I have been mulling over all these pieces for a while, thinking about what a better system would do. There are certain cases, like aggregation, where some kind of caching is probably necessary for performance. Arguably maybe not for single files.
Anyway, you might try running without caching. If you are able to get any timings for the two cases (with and without), I would be very interested.
Cheers, Jon On Mon, Sep 28, 2009 at 5:57 PM, John Caron <caron@xxxxxxxxxxxxxxxx> wrote:Jon Blower wrote:Dear all (esp. John), I think I have been misunderstanding the intended behaviour of the cache of NetCDF files (the one that's initialized with NetcdfDataset.initNetcdfFileCache()). What is the behaviour of the "periodic cleanup"? Does it remove elements from memory only when the number of elements in memory exceeds a certain number? Or does it also remove elements based on their age?every n minutes it removes files using LRU.The behaviour I would like is the latter - i.e. I would like to be able to guarantee that the elements in the cache are no older than, say, five minutes. But this isn't the behaviour I'm seeing: elements seem to persist essentially for ever. So perhaps the former behaviour is the correct one? In which case, how would you recommend I implement the latter behaviour?ill probably move to ehcache (version 4.2?), which i think allows this. ill have to look at the code for a workaround...(Rich and Clare - I think this might be causing the caching problems you are seeing with ncWMS.) Thanks, Jon
netcdf-java archives: