[thredds] Max file size for WCS getCoverage response

• To: <thredds@xxxxxxxxxxxxxxxx>
• Subject: [thredds] Max file size for WCS getCoverage response
• From: Oliver Clements <olcl@xxxxxxxxx>
• Date: Mon, 16 Dec 2013 15:16:39 +0000

Hi,

We have a large dataset which in total is around 27Gb for a single
variable across the complete temporal and spatial domain.  We have found
that if we request the complete data set through WCS we simply get a
HTTP 400 error with a serviceException message of 'Unknown problem'
(full output below).  When we looked in the Thredds logs we find an
error stating that variable cannot be over 4294967292 bytes.  Is this
size configurable? if not is there any sensible way to detect this error
in the client? We would like to be able to catch this error and then
split the get coverage request up into more manageable chunks.

Thanks for an help

Olly Clements

CURL output :

curl -i
"http://earthserver.pml.ac.uk/thredds/wcs/CCI-v095-M?crs=OGC%3ACRS84&service=WCS&format=NetCDF3&request=GetCoverage&version=1.0.0&coverage=chlor_a&time=1997-12-01%2F2013-12-31";
HTTP/1.1 400 Bad Request
Date: Mon, 16 Dec 2013 10:34:34 GMT
Content-Type: application/vnd.ogc.se_xml;charset=ISO-8859-1
Connection: close
Transfer-Encoding: chunked

<?xml version="1.0" encoding="UTF-8"?>
<ServiceExceptionReport xmlns="http://www.opengis.net/ogc"; version="1.2.0">
  <ServiceException>Unknown problem</ServiceException>
</ServiceExceptionReport>

Thredds.log :

2013-12-16T09:43:52.021 +0000 [ 248161287][   90503] INFO  -
thredds.server.wcs.WCSServlet - Remote host: 192.171.164.51 - Request:
"GET
/thredds/wcs/CCI-v095-M?crs=OGC%3ACRS84&service=WCS&format=NetCDF3&request=GetCoverage&version=1.0.0&coverage=chlor_a&time=1997-12-01%2F2013-12-31
HTTP/1.1"
2013-12-16T09:43:52.021 +0000 [ 248161287][   90503] DEBUG -
thredds.servlet.DatasetHandler - DatasetHandler wants /CCI-v095-M
2013-12-16T09:43:52.021 +0000 [ 248161287][   90503] DEBUG -
thredds.servlet.DatasetHandler -   -- DatasetHandler found NcmlDataset=
CCI-v095-M
2013-12-16T09:43:52.437 +0000 [ 248161703][   90503] DEBUG -
thredds.wcs.v1_0_0_1.WcsCoverage - writeCoverageDataToFile():
ncFile=/opt/tomcat/apache-tomcat-7.0.29/content/thredds/cache/wcs/WCS4099895692493199293.nc
2013-12-16T09:43:52.438 +0000 [ 248161704][   90503] INFO  -
ucar.nc2.dt.grid.NetcdfCFWriter - Request size = 27620 Mbytes
2013-12-16T09:43:52.438 +0000 [ 248161704][   90503] ERROR -
thredds.server.wcs.v1_0_0_1.WcsHandler - Unknown problem.
java.lang.IllegalArgumentException: Variable size in bytes 27620352000
may not exceed 4294967292
    at
ucar.nc2.NetcdfFileWriteable.addVariable(NetcdfFileWriteable.java:431)
    at
ucar.nc2.NetcdfFileWriteable.addVariable(NetcdfFileWriteable.java:372)
    at ucar.nc2.FileWriter.writeVariable(FileWriter.java:581)
    at ucar.nc2.FileWriter.writeVariables(FileWriter.java:600)
    at ucar.nc2.dt.grid.NetcdfCFWriter.makeFile(NetcdfCFWriter.java:189)
    at
thredds.wcs.v1_0_0_1.WcsCoverage.writeCoverageDataToFile(WcsCoverage.java:280)
    at
thredds.wcs.v1_0_0_1.GetCoverage.writeCoverageDataToFile(GetCoverage.java:169)
    at thredds.server.wcs.v1_0_0_1.WcsHandler.handleKVP(WcsHandler.java:125)
    at thredds.server.wcs.WCSServlet.doGet(WCSServlet.java:251)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
    at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
    at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at
thredds.servlet.filter.RequestQueryFilter.doFilter(RequestQueryFilter.java:122)
    at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at
thredds.servlet.filter.RequestPathFilter.doFilter(RequestPathFilter.java:105)
    at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225)
    at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
    at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
    at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
    at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
    at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
    at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
    at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
    at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:200)
    at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:585)
    at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)
2013-12-16T09:43:52.439 +0000 [ 248161705][   90503] INFO  -
thredds.server.wcs.v1_0_0_1.WcsHandler - Request Completed - 400 - -1 - 418


Latest news: www.pml.ac.uk and @PlymouthMarine

Plymouth Marine Laboratory (PML) is a company limited by guarantee registered 
in England & Wales, company number 4178503. Registered Charity No. 1091222. 
Registered Office: Prospect Place, The Hoe, Plymouth  PL1 3DH, UK. 

This message is private and confidential. If you have received this message in 
error, please notify the sender and remove it from your system. You are 
reminded that e-mail communications are not secure and may contain viruses; PML 
accepts no liability for any loss or damage which may be caused by viruses.