NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

Re: [thredds] Running out of memory on a joinExisting aggregation data set

Xmx likely needs to be more in the range of 1200Mb in our experience.

-Roy

On Jan 12, 2009, at 12:27 PM, Ken Tanaka wrote:


John and Roy,
It looks like the current Xmx memory setting is 256 MB for the tomcat process. I'll arrange to have the memory increased, but we were concerned that we would still run out of memory--just at a higher number of files. Earlier posts indicated that I should be able to aggregate a large number of files (over 2000), so I wasn't sure if I was configuring the server correctly. Should I delve into caching or types of performance tuning? We would like to add a file once a day and to an unlimited Time coordinate, our joinExisting dimension, indefinitely.
We were thinking of moving a large number of Global Attributes (metadata) out of the files and into an NcML file since most of it is the same (constant) in all the files. We felt that it made sense to place the aggregation and attributes into the NcML file rather than clutter the catalog.xml in this case. Will this save memory to abstract out the repeated Global Attributes and provide them in a single NcML wrapper? 

-Ken

John Caron wrote:


Hi Ken:

How much memory are you giving the JVM?
These kinds of aggregations should not consume memory, but there are some bugs in there.
You configuration looks ok, although you could embed the ncml directly into the catalog:
<dataset name="SST Aerosol Aggregation NOAA-18(N)" ID="SST-Aerosol- NN-Agg" 
          urlPath="sst-aerosol-nn-aggregation">
   <metadata inherited="true">
     <serviceName>multiple</serviceName>
     <dataType>Grid</dataType>
   </metadata>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2 " > 
     <attribute name="myAttribute" value="NN-attribute" />
     <aggregation dimName="Time" type="joinExisting">
       <scan location="/data1/thredds/sst/Aerosol_NN_test/"
             suffix=".nc" />
     </aggregation>
    </netcdf>

  </dataset>
I dont think that should matter, but if its easy, why dont you try that? 

Meanwhilie ill try to reproduce the problem.

Ken Tanaka wrote:


Hi,
I'm encountering "java.lang.OutOfMemoryError: Java heap space" errors in 
aggregating netcdf files. The messages are appearing in the
tomcat5/content/thredds/logs/threddsServlet.log and the web page will 
return a web page showing
Error { code = -1; message = "Server Error on dataset sst-aerosol- nl-aggregation"; }; 


instead of the normal OPENDAP access page (
http://www.ngdc.noaa.gov/thredds/dodsC/sst-aerosol-nl-aggregation.html
). This page usually returns normally for a few requests, but once the OutOfMemoryError occurs the service is down until I restart tomcat. I did try an upgrade to the latest THREDDS warfile, 3.16.48.0, but still 
see this error. The TDS is running on Red Hat Enterprise Linux WS
release 3 (Taroon Update 9). The java.vm.version = 1.5.0_06-b05, and
tomcat version is 5.5.17.

The files we are serving up are SST Aerosol products, with a Time
coordinate, with a single time slice per file. The typical filesize is 3.6MB. We have quite a bit of metadata as Global Attributes in each, but I think we can abstract that out with an NcML wrapper file. Here is the 
wrapper file, which is in the same directory with the data files:

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2 " > 
     <attribute name="myAttribute" value="NN-attribute" />
     <aggregation dimName="Time" type="joinExisting">
       <scan location="/data1/thredds/sst/Aerosol_NN_test/"
             suffix=".nc" />
     </aggregation>

</netcdf>
And here is the portion of the catalog.xml that uses this aggregation: 

<!-- Aerosol NN, joinExisting with ncml wrapper
-->
<dataset name="SST Aerosol Aggregation NOAA-18(N)" ID="SST- Aerosol-NN-Agg" 
          urlPath="sst-aerosol-nn-aggregation">
   <metadata inherited="true">
     <serviceName>multiple</serviceName>
     <dataType>Grid</dataType>
   </metadata>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2 " location="/data1/thredds/sst/Aerosol_NN_test/ Aerosol_NN_test.ncml" > 
    </netcdf>

  </dataset>



Do you see any troubles with my configuration? Earlier posts to this
list indicated that there should not be a problem with aggregating a
large number of files. This particular aggregation is 245 files. I
definitely get errors right away if I also set up separate but similar 
aggregation with 135 files in the same catalog.

Thanks for any suggestions,
Ken



_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/



--
= Enterprise Data Services Division ===============
| CIRES, National Geophysical Data Center / NOAA  |
| 303-497-6221                                    |
= Ken.Tanaka@xxxxxxxx =============================
_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/


**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA." 
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
  • 2009 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: