Re: [thredds] Nightly restart of thredds-webapplication? Too many open files?

• To: Ethan Davis <edavis@xxxxxxxxxxxxxxxx>
• Subject: Re: [thredds] Nightly restart of thredds-webapplication? Too many open files?
• From: Heiko Klein <Heiko.Klein@xxxxxx>
• Date: Mon, 07 Sep 2009 10:15:55 +0200

Hi Ethan,

my specs have been wrong, we are already running tomcat 6.0.20.

'INFO: Destroying Spring FrameworkServlet '...''

Heiko


Ethan Davis wrote:

Hi Heiko,

An automatic, nightly shutdown of the TDS is not normal behavior. We
keep the TDS up for long periods of time and I've never seen a
spontaneous shutdown of the TDS. I'm wondering if the "too many open
files" exceptions are causing the problem. Do you get those exceptions
before every shutdown?

Also, what version of Tomcat 6 are you running? From this page

http://www.nabble.com/NIO-Connector:-Too-many-open-files-td23655733.html

it looks like there was a file descriptor leak in nio-connector up
through Tomcat 6.0.18. If you are running Tomcat 6.0.18, can you upgrade
to Tomcat 6.0.20 and see if that helps with the "too many open files"
exception.

Ethan

Heiko Klein wrote:

Hi,

I've just had some problems with our tomcat server running thredds, and
I would like to find out more about the reasons for it. We're running
tomcat 6 (apache proxy, nio-connector) and thredds 4.0.25 on linux 32bit
with java6.

Each night, between 12:25 and 12:30 AM, the thredds webapplication seems
to be restarted internally (localhost.2009-09-04.log):
Sep 4, 2009 12:29:04 AM org.apache.catalina.core.ApplicationContext log
INFO: Destroying Spring FrameworkServlet 'catalogService'
Sep 4, 2009 12:29:04 AM org.apache.catalina.core.ApplicationContext log
INFO: Destroying Spring FrameworkServlet 'root'
...
Sep 4, 2009 12:29:11 AM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring FrameworkServlet 'cdmRemote'
Sep 4, 2009 12:29:11 AM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring FrameworkServlet 'admin'
Sep 4, 2009 12:29:11 AM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring FrameworkServlet 'wms'

Is this the normal behaviour. Why is the restart needed? Where is it
scheduled?

Some days ago, this restart didn't work and crashed tomcat:
...
Sep 1, 2009 12:30:14 AM org.apache.catalina.core.ApplicationContext log
INFO: Destroying Spring FrameworkServlet 'wms'
Sep 1, 2009 12:30:14 AM org.apache.catalina.core.ApplicationContext log
INFO: Shutting down log4j
Sep 1, 2009 12:30:15 AM org.apache.catalina.core.StandardContext
listenerStop
SEVERE: Exception sending context destroyed event to listener instance
of class thredds.server.config.TdsConfigContextListener
java.lang.NoClassDefFoundError: org/apache/log4j/helpers/NullEnumeration
        at org.apache.log4j.Category.getAllAppenders(Category.java:413)
        at
org.apache.log4j.Category.closeNestedAppenders(Category.java:226)
        at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:448)
        at org.apache.log4j.LogManager.shutdown(LogManager.java:238)
        at
org.springframework.util.Log4jConfigurer.shutdownLogging(Log4jConfigurer.java:117)

...
INFO: Closing Spring root WebApplicationContext


We've also experience some 'Too many open files' Exception around that
time:

SEVERE: Socket accept failed
java.io.IOException: Too many open files
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
        at
org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1266)
        at java.lang.Thread.run(Thread.java:619)

and

Sep 1, 2009 12:30:16 AM org.apache.catalina.startup.ContextConfig
defaultWebConfig
SEVERE: Missing default web.xml, using application web.xml only
conf/web.xml /metno/eksternweb/thredds/apache-tomcat-6.0.20/conf/web.xml
java.io.FileNotFoundException:
/metno/eksternweb/thredds/apache-tomcat-6.0.20/conf/web.xml (Too many
open files)


Has somebody else seen the (Too many open files) with thredds, or is
something in our configuraton (machine/tomcat) wrong? For more
information , please see the attached log-files.

Best regards,

Heiko

--
Dr. Heiko Klein                              Tel. + 47 22 96 32 58
Development Section / IT Department          Fax. + 47 22 69 63 55
Norwegian Meteorological Institute           http://www.met.no
P.O. Box 43 Blindern  0313 Oslo NORWAY

• References:
  • [thredds] Nightly restart of thredds-webapplication? Too many open files?
    • From: Heiko Klein
  • Re: [thredds] Nightly restart of thredds-webapplication? Too many open files?
    • From: Ethan Davis