[netcdf-java] NetCDF GridSubsetService Out of memory errors

• To: <netcdf-java@xxxxxxxxxxxxxxxx>, <thredds@xxxxxxxxxxxxxxxx>
• Subject: [netcdf-java] NetCDF GridSubsetService Out of memory errors
• From: <Trevor.Daniels@xxxxxxxx>
• Date: Tue, 30 Oct 2007 13:15:38 +1100

 
Hi,

I'm getting out of memory errors trying to aggregate some rainfall data
files (each 2.5Mb) using the netcdf grid subsetting service in thredds
(3.16.19) with the following..

<dataset name="Rainfall" ID="RainAggregated"
urlPath="content/bilo/aggregatedData.nc">
 <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
        <aggregation dimName="time" type="joinNew">
          <variableAgg name="rain"/>
          <scan location="c:\data\bilo\2007\" suffix=".nc"
dateFormatMark="#yyyyMMdd" />
        </aggregation>
      </netcdf>

Generally the aggregation works for producing outputs of up to about
50Mb, whereby anything greater generates the following stack trace on
multiple pc's...

Oct 29, 2007 3:34:48 PM org.apache.catalina.core.StandardWrapperValve
invoke
SEVERE: Servlet.service() for servlet GridSubsetService threw exception
java.lang.OutOfMemoryError: Java heap space
 at ucar.ma2.ArrayFloat.<init>(ArrayFloat.java:86)
 at ucar.ma2.ArrayFloat$D3.<init>(ArrayFloat.java:291)
 at ucar.ma2.ArrayFloat$D3.<init>(ArrayFloat.java:283)
 at ucar.ma2.ArrayFloat.factory(ArrayFloat.java:50)
 at ucar.ma2.ArrayFloat.factory(ArrayFloat.java:36)
 at ucar.ma2.Array.factory(Array.java:130)
 at ucar.ma2.Array.factory(Array.java:79)
 at ucar.nc2.ncml.Aggregation.read(Aggregation.java:666)
 at ucar.nc2.dataset.VariableDS._read(VariableDS.java:277)
 at ucar.nc2.Variable.read(Variable.java:603)
 at ucar.nc2.Variable.read(Variable.java:556)
 at ucar.nc2.FileWriter.copySome(FileWriter.java:287)
 at ucar.nc2.FileWriter.copyVarData(FileWriter.java:225)
 at ucar.nc2.FileWriter.finish(FileWriter.java:459)
 at ucar.nc2.dt.grid.NetcdfCFWriter.makeFile(NetcdfCFWriter.java:198)
 at thredds.server.ncSubset.GridServlet.sendFile(GridServlet.java:395)
 at
thredds.server.ncSubset.GridServlet.processGrid(GridServlet.java:354)
 at thredds.server.ncSubset.GridServlet.doGet(GridServlet.java:148)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
 at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:290)
 at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:206)
 at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv
e.java:233)
 at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValv
e.java:175)
 at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(Authenticator
Base.java:433)
 at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java
:128)
 at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java
:102)
 at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.
java:109)
 at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:2
63)
 at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:84
4)
 at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(
Http11Protocol.java:584)
 at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
        
Any suggestions on how to resolve or further diagnose?

Thanks for help,

Trevor

• Follow-Ups:
  • Re: [netcdf-java] [thredds] NetCDF GridSubsetService Out of memory errors
    • From: Roy Mendelssohn