NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Hi John, I find it very encouraging that you can run with 20,000 files.We've been running with 1024 MB allocated to the JVM and haven't had anymore trouble, so I think the memory was a problem. On investigating further, it appears that no memory was specified before, so the trouble was occurring with a 64 MB JVM. The 256 MB was for a tomcat instance serving a different purpose.
I'll leave the individual files alone since abstracting out the common attributes won't save us that much.
Thanks for your help John and Roy. -Ken John Caron wrote:
Hi Ken: Ken Tanaka wrote:John and Roy, It looks like the current Xmx memory setting is 256 MB for the tomcat process. I'll arrange to have the memory increased,
...
Im currently testing with 20,000 files, which it is handling ok. There is a small amount of memory retained for each file, which i am currently trying to minimize. I think you'll find that if you can give the server 1M or more, your case will work fine. You will also see significant performance by using java 1.6.Meanwhile, i will try to keep things as sleek as possible. We will eventually get object caching working, to ensure there will be no memory limits at all. New development is being done only on the 4.0 branch. Its stable enough to start testing with it if you want to compare performance, etc. Look for it on the TDS home page: http://www.unidata.ucar.edu/projects/THREDDS/tech/TDS.htmlWe were thinking of moving a large number of Global Attributes (metadata) out of the files and into an NcML file since most of it is the same (constant) in all the files. We felt that it made sense to place the aggregation and attributes into the NcML file rather than clutter the catalog.xml in this case. Will this save memory to abstract out the repeated Global Attributes and provide them in a single NcML wrapper?There wont be any real memory savings by factoring out attributes into NcML. There will be some minor speedup if you actually remove them from the netcdf files. For now, I wouldn't try to do such optimizations. Let me know if your problem goes away with a larger memory size. _______________________________________________ thredds mailing list thredds@xxxxxxxxxxxxxxxxFor list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/
-- = Enterprise Data Services Division =============== | CIRES, National Geophysical Data Center / NOAA | | 303-497-6221 | = Ken.Tanaka@xxxxxxxx =============================
thredds archives: