NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
The server is CentOS 5.7, with 48 Gb main memory using Sun/Oracle Java 1.6 and tomcat 6. Our catalog is pretty small, with only a couple hundred datasets. With such a small catalog I setup our machine to restart tomcat each night, which only takes about 2 seconds. Spiking 12 CPUs to 100% for over a minute is still curious and bogs down response time when it occurs. Hopefully the nightly restarts will prevent TDS from becoming completely unresponsive. On Dec 12, 2013, at 9:47 AM, Gerry Creager - NOAA Affiliate <gerry.creager@xxxxxxxx> wrote: > Jay, > > What OS, release and Tomcat version are you running? I've seen a similar > issue on another piece of software (Ramadda). Since I've seen this behavior > with the standalone server and the tomcat-based server, I'm beginning to > suspect my Java installation, but have not had sufficient time to investigate > yet. > > There may be an OS correlation here, so I'm interested. I'm running RHEL 6 > and the various updated flavors of OpenJDK and Tomcat6. > > gerry > > > > > On Wed, Dec 11, 2013 at 4:33 PM, Jay Alder <alderj@xxxxxxxxxxxxxxxxxxxx> > wrote: > Hi, we’ve recently released a web application that uses TDS for mapping, > which is getting a lot of traffic. At one point the server stopped responding > altogether, which is a major problem. A quick restart of tomcat got it going > again, so I’m starting to dig into the logs. We normally get the GET / > request complete behavior, but occasionally we’ll have: > > GET …url… > GET …url… > GET …url… > GET …url… > GET …url… > GET …url… > GET …url… > GET …url… > > meanwhile having a 100% CPU spike (with 12 CPUs) for a minute or more > > request compete > request compete > request compete > request cancelled by client > request cancelled by client > request compete > request compete > > While watching the logs the few times I’ve seen this occur it seems to pull > out of it ok. However the time the server failed, requests were never > returned. From the logs, requests came in for roughly 40 minutes without > being completed. Unfortunately do to the high visibility we started to get > emails from users and the press about the application no longer working. > > Has anyone experienced this before and/or can you give guidance on how to > diagnose or prevent this? > > Here are some config settings: > CentOS 5.7 > Java 1.6 > TDS 4.3.17 > only WMS is enabled > Java -Xmx set to 8Gb (currently taking 5.3, the dataset is 600 Gb of > 30-arcsecond grids for the continental US, 3.4 Gb per file) > For better or worse we are configured to use 2 instances of TDS to keep the > catalogs and configuration isolated. I’m not sure if this matters, but I > didn’t want to omit it. Since it is a live server I can’t easily change to > the preferred proxy configuration. > > I am trying not to panic yet. However, if the server goes unresponsive again, > staying calm may no longer be an option. > > Jay Alder > US Geological Survey > Oregon State University > 104 COAS Admin Building > Office Burt Hall 166 > http://ceoas.oregonstate.edu/profile/alder/ > > > _______________________________________________ > thredds mailing list > thredds@xxxxxxxxxxxxxxxx > For list information or to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/ > > > > -- > Gerry Creager > NSSL/CIMMS > 405.325.6371 > ++++++++++++++++++++++ > “Big whorls have little whorls, > That feed on their velocity; > And little whorls have lesser whorls, > And so on to viscosity.” > Lewis Fry Richardson (1881-1953) Jay Alder US Geological Survey Oregon State University 104 COAS Admin Building Office Burt Hall 166 http://ceoas.oregonstate.edu/profile/alder/ Jay Alder US Geological Survey Oregon State University 104 COAS Admin Building Office Burt Hall 166 http://ceoas.oregonstate.edu/profile/alder/
thredds archives: