NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
And it is this initial indexing situation that I am currently dealing with.I have been triggering this initial scan by clicking on the services for the aggregated dataset. Is there another way to perform the initial indexing of netCDF aggregations (like is done with GRIB Collections) besides clicking on a service link?
Also, I assume that the scouring of NetcdfFileCache would not remove this index file from cache/agg, correct? Otherwise users would be in for a long wait each time they click on an aggregated service. According to http://www.unidata.ucar.edu/software/thredds/current/tds/reference/ThreddsConfigXMLFile.html, the cache/agg dir is only for joinExisting. I'm trying to use Union right now.
-kevin. On 10/25/13 12:20 PM, Michael McDonald wrote:
John et al,Aggregations are not supposed to leave files open. If they are, thats a bug. The number of open files should typically be less than 1000, no matter how big your aggregations are. It is dependent on the file cache sizes, see:We set our NetcdfFileCache to match what we set our ulimit max # of open files to (1mil). The hope of this is that as many files as possible are cached in memory by the server, for quicker catalog scans/requests. We never even approach 1mil concurrently open files. <NetcdfFileCache> <minFiles>1000</minFiles> <maxFiles>1000000</maxFiles> <scour>24 hours</scour> </NetcdfFileCache> It was also my understanding that ALL of the files in a particular dataset scan need to be touched/opened/checked before being folded into an aggregation (this is upon initial scan for saving into the cache/agg folder). After this initial indexing, the average # of open files will drop off since queries are only picking/choosing a few files from this catalog, but you still need to have all the proper system limits bumped up in order for this initial scan/cache to ever successfully complete in a timely fashion. correct/incorrect? /mike
-- Kevin Manross NCAR/CISL/Data Support Section Phone: (303)-497-1218 Email:manross@xxxxxxxx <mailto:manross@xxxxxxxx> Web:http://rda.ucar.edu
thredds archives: