NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Hi Brian:1) on startup, the TDS needs to scan your directories to see whats in the collection. once scanned, it wont do it again.
2) FMRC will be refactored (probably in 4.4) to write the results to disk, and startup=nocheck will use those if available. until then, it has to get scanned
3) i do have an experimental ControllerCache class that will cache the OS file scan. It wasnt reliable enough across platforms for files that were changing, so i disabled. If you want to play with it, contact me. it would be most useful for static collections.
How big is the collection, and how long does it take to scan? John On 7/31/2013 9:45 AM, Brian McKenna wrote:
I’m having some issues with THREDDS and a very high number of
interrupts/context switches killing the server’s performance. Increasing
the LOG level shows the high switching related to
thredds.filesystem.ControllerOS. As soon as the messages pasted below
begin, the interrupt/context switch number jumps from <100 to nearly
20,000. CPU and memory do not appear to be a limiting factor, their
loads remain low. The filesystem is mounted via NFS, but read/write
speeds (via dd) seem fine, greater than 150 megabytes/s. Those speeds
are of course without the massive context switching overhead.
Despite <update startup ="nocheck" /> being declared (see catalog
below), ControllerOS seems to be called in a loop somewhere and appears
to be listing every directory and file within and repeats itself once
complete.
Is there a way to avoid such heavy context switching related to this
listing of files, possibly by consolidating the files to a single
location, or disabling the ‘listing’ being done by ControllerOS all
together?
Thanks,
Brian McKenna
threddsServlet.log
-----------------------
2013-07-31T09:06:53.614 -0400 [ 11067][ ] DEBUG -
thredds.filesystem.ControllerOS - List Directory
/data/maracoos/ensemble/WRF_ensemble_small
2013-07-31T09:06:53.630 -0400 [ 11083][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble_small/2012.02.10.00
2013-07-31T09:06:53.633 -0400 [ 11086][ ] DEBUG -
thredds.filesystem.ControllerOS - List Directory
/data/maracoos/ensemble/WRF_ensemble
2013-07-31T09:06:53.634 -0400 [ 11087][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble_small/2012.06.03.00
2013-07-31T09:06:53.638 -0400 [ 11091][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble_small/2012.09.30.00
2013-07-31T09:06:53.641 -0400 [ 11094][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble_small/2012.03.05.00
2013-07-31T09:06:53.645 -0400 [ 11098][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble_small/2012.03.14.12
2013-07-31T09:06:53.648 -0400 [ 11101][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble_small/2012.10.05.12
2013-07-31T09:06:53.651 -0400 [ 11104][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble_small/2012.10.01.00
2013-07-31T09:06:53.653 -0400 [ 11106][ ] DEBUG -
thredds.filesystem.ControllerOS - List Directory
/data/maracoos/ensemble/AKQ_WRF
2013-07-31T09:06:53.659 -0400 [ 11112][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble/2012.02.10.00
2013-07-31T09:06:53.660 -0400 [ 11113][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble_small/2012.06.01.00
2013-07-31T09:06:53.664 -0400 [ 11117][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble/2012.06.03.00
2013-07-31T09:06:53.664 -0400 [ 11117][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble_small/2012.11.16.00
2013-07-31T09:06:53.668 -0400 [ 11121][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble/2011.12.19.00
2013-07-31T09:06:53.668 -0400 [ 11121][ ] DEBUG -
thredds.filesystem.ControllerOS - File
/data/maracoos/ensemble/WRF_ensemble_small/2012.05.04.12
featureCollection from catalog
---------------------
<featureCollection featureType="FMRC" name="Ensemble WRF (large grid)
Forecast Aggregation Collection" harvest="true"
path="WRF-Large/Agg-Collection.nc">
<metadata inherited="true">
<serviceName>agg</serviceName>
<dataFormat>NetCDF</dataFormat>
</metadata>
<collection
spec="/data/maracoos/ensemble/WRF_ensemble/**/#yyyy.MM.dd.HH#.*\.nc$"
name=" WRF_Large_Ensemble" olderThan="5 min" />
<update startup="nocheck" trigger="allow" />
<fmrcConfig regularize="true" datasetTypes="Best Files Runs" />
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
<variable name="lat" orgName="g0_lat_0">
<attribute name="standard_name" value="latitude" />
</variable>
<variable name="lon" orgName="g0_lon_1">
<attribute name="standard_name" value="longitude" />
</variable>
<variable name="u" orgName="U_GRD_GDS0_HTGL_10">
<attribute name="standard_name" value="eastward_wind_velocity" />
<attribute name="coordinates" value="time lat lon" />
</variable>
<variable name="v" orgName="V_GRD_GDS0_HTGL_10">
<attribute name="standard_name" value="northward_wind_velocity" />
<attribute name="coordinates" value="time lat lon" />
</variable>
<variable name="pressure" orgName="PRMSL_GDS0_SFC_10">
<attribute name="coordinates" value="time lat lon" />
</variable>
<variable name="temperature" orgName="TMP_GDS0_HTGL_10">
<attribute name="coordinates" value="time lat lon" />
</variable>
<variable name="humidity" orgName="R_H_GDS0_HTGL_10">
<attribute name="coordinates" value="time lat lon" />
</variable>
</netcdf>
</featureCollection>
_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
thredds archives: