Re: [thredds] NetCDF GridSubsetService Out of memory errors

• To: Trevor.Daniels@xxxxxxxx
• Subject: Re: [thredds] NetCDF GridSubsetService Out of memory errors
• From: Roy Mendelssohn <Roy.Mendelssohn@xxxxxxxx>
• Date: Mon, 29 Oct 2007 19:25:47 -0700

-Roy M.

On Oct 29, 2007, at 7:15 PM, Trevor.Daniels@xxxxxxxx wrote:

Hi,

I'm getting out of memory errors trying to aggregate some rainfall data

files (each 2.5Mb) using the netcdf grid subsetting service in thredds
(3.16.19) with the following..

<dataset name="Rainfall" ID="RainAggregated"
urlPath="content/bilo/aggregatedData.nc">

<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ ncml-2.2">

        <aggregation dimName="time" type="joinNew">
          <variableAgg name="rain"/>
          <scan location="c:\data\bilo\2007\" suffix=".nc"
dateFormatMark="#yyyyMMdd" />
        </aggregation>
      </netcdf>

Generally the aggregation works for producing outputs of up to about
50Mb, whereby anything greater generates the following stack trace on
multiple pc's...

Oct 29, 2007 3:34:48 PM org.apache.catalina.core.StandardWrapperValve
invoke

SEVERE: Servlet.service() for servlet GridSubsetService threw exception

java.lang.OutOfMemoryError: Java heap space
 at ucar.ma2.ArrayFloat.<init>(ArrayFloat.java:86)
 at ucar.ma2.ArrayFloat$D3.<init>(ArrayFloat.java:291)
 at ucar.ma2.ArrayFloat$D3.<init>(ArrayFloat.java:283)
 at ucar.ma2.ArrayFloat.factory(ArrayFloat.java:50)
 at ucar.ma2.ArrayFloat.factory(ArrayFloat.java:36)
 at ucar.ma2.Array.factory(Array.java:130)
 at ucar.ma2.Array.factory(Array.java:79)
 at ucar.nc2.ncml.Aggregation.read(Aggregation.java:666)
 at ucar.nc2.dataset.VariableDS._read(VariableDS.java:277)
 at ucar.nc2.Variable.read(Variable.java:603)
 at ucar.nc2.Variable.read(Variable.java:556)
 at ucar.nc2.FileWriter.copySome(FileWriter.java:287)
 at ucar.nc2.FileWriter.copyVarData(FileWriter.java:225)
 at ucar.nc2.FileWriter.finish(FileWriter.java:459)
 at ucar.nc2.dt.grid.NetcdfCFWriter.makeFile(NetcdfCFWriter.java:198)
 at thredds.server.ncSubset.GridServlet.sendFile(GridServlet.java:395)
 at
thredds.server.ncSubset.GridServlet.processGrid(GridServlet.java:354)
 at thredds.server.ncSubset.GridServlet.doGet(GridServlet.java:148)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
 at

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter (Applica

tionFilterChain.java:290)
 at

org.apache.catalina.core.ApplicationFilterChain.doFilter (ApplicationFilt

erChain.java:206)
 at

org.apache.catalina.core.StandardWrapperValve.invoke (StandardWrapperValv

e.java:233)
 at

org.apache.catalina.core.StandardContextValve.invoke (StandardContextValv

e.java:175)
 at

org.apache.catalina.authenticator.AuthenticatorBase.invoke (Authenticator

Base.java:433)
 at

org.apache.catalina.core.StandardHostValve.invoke (StandardHostValve.java

:128)
 at

org.apache.catalina.valves.ErrorReportValve.invoke (ErrorReportValve.java

:102)
 at

org.apache.catalina.core.StandardEngineValve.invoke (StandardEngineValve.

java:109)
 at

org.apache.catalina.connector.CoyoteAdapter.service (CoyoteAdapter.java:2

63)
 at

org.apache.coyote.http11.Http11Processor.process (Http11Processor.java:84

4)
 at

org.apache.coyote.http11.Http11Protocol $Http11ConnectionHandler.process(

Http11Protocol.java:584)
 at

org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java: 447)

        
Any suggestions on how to resolve or further diagnose?

Thanks for help,

Trevor

_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx

For list information or to unsubscribe, visit: http:// www.unidata.ucar.edu/mailing_lists/

**********************

**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division 
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."

• References:
  • [thredds] NetCDF GridSubsetService Out of memory errors
    • From: Trevor.Daniels