NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Hi all: Aggregations arent supposed to leave files open. If they are, thats a bug. The number of open files should typically be less than 1000, no matter how big your aggregations are. It is dependent on the file cache sizes, see: http://www.unidata.ucar.edu/software/thredds/v4.4/tds/reference/ThreddsConfigXMLFile.html#FileCache So if you are seeing lots of open files, we have a file leak i think. You can verify by turning your file cache to 0 or very low (10?). Then watch the pfiles output to see whats happening. John On 10/24/2013 3:15 PM, Kevin Manross wrote: > > This is very helpful Mike - thank you very much for the insight and your > experience. > > This will be useful information for requesting the ulimit increase from > our sysadmins! > > -kevin. > > > On 10/24/13 3:09 PM, Michael McDonald wrote: >> Kevin, >> We needed to add a script to "/etc/profile.d" that increases the stock >> CentOS/RHEL 6.x number of open files. Change $USER == "tomcat" to >> whatever you run tomcat/apache as (see below). Also assumes that the >> user is running the sh/bash shell. >> >> # cat /etc/profile.d/thredds.sh >> export TOMCAT_HOME="/usr/share/tomcat6" >> if [ $USER == "tomcat" ]; then ulimit -n 1000000; fi >> >> >> Our THREDDS servers providing only OPENDAP requests are averaging 40k >> open files. Our THREDDS servers providing only NCSS requests are >> averaging 15k open files. We have a lot of files in some of our hycom >> datasets. >> >> /mike >> >> On Thu, Oct 24, 2013 at 1:50 PM, Kevin Manross<manross@xxxxxxxx> wrote: >>> Hi there, >>> >>> I am trying to aggregate netcdf files for some rather large datasets with >>> tomcat6.0.36/TDS4.19 - 20131008.1358 and am running into a "Too many open >>> files" error. >>> >>> My catalog scan is: >>> >>> <dataset name="ASR 30 km 2d surface analysis" >>> ID="ds631.0_anl_2d-AGG" >>> urlPath="ds631.0/9/best" >>> harvest="true"> >>> <serviceName>all</serviceName> >>> <netcdf >>> xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";> >>> <aggregation dimName="time" type="Union"> >>> <scan location="/glade/p/rda/data/ds631.0/asr30km.anl.2d/" >>> suffix=".nc" subdirs="false"/> >>> </aggregation> >>> </netcdf> >>> </dataset> >>> >>> which contains about 4020 files and totals 470G. >>> >>> The message that the server sends when trying to access the OPeNDAP service >>> is: >>> >>> >>> Error { >>> code = 404; >>> message = >>> "/glade/p/rda/data/ds631.0/asr30km.anl.2d/asr30km.anl.2d.20010619.nc (Too >>> many open files)"; >>> }; >>> >>> >>> >>> When trying the CdmRemote service catalina.out repeatedly reads: >>> >>> Oct 24, 2013 11:20:39 AM org.apache.jk.common.ChannelSocket >>> acceptConnections >>> WARNING: Exception executing accept >>> java.net.SocketException: Too many open files >>> at java.net.PlainSocketImpl.socketAccept(Native Method) >>> at >>> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:375) >>> at java.net.ServerSocket.implAccept(ServerSocket.java:478) >>> at java.net.ServerSocket.accept(ServerSocket.java:446) >>> at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:311) >>> at >>> org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:668) >>> at >>> org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:879) >>> at >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690) >>> at java.lang.Thread.run(Thread.java:679) >>> >>> >>> The cache settings in my threddsConfig.xml are the default settings. >>> >>> Have I overlooked something? >>> >>> -kevin. >>> >>> >>> -- >>> Kevin Manross >>> NCAR/CISL/Data Support Section >>> Phone: (303)-497-1218 >>> Email:manross@xxxxxxxx >>> Web:http://rda.ucar.edu >>> >>> _______________________________________________ >>> thredds mailing list >>> thredds@xxxxxxxxxxxxxxxx >>> For list information or to unsubscribe, visit: >>> http://www.unidata.ucar.edu/mailing_lists/ > > -- > Kevin Manross > NCAR/CISL/Data Support Section > Phone: (303)-497-1218 > Email:manross@xxxxxxxx <mailto:manross@xxxxxxxx> > Web:http://rda.ucar.edu > > > _______________________________________________ > thredds mailing list > thredds@xxxxxxxxxxxxxxxx > For list information or to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/ >
thredds archives: