[thredds] thredds.filesystem.ControllerOS massive context switching

• To: "thredds@xxxxxxxxxxxxxxxx" <thredds@xxxxxxxxxxxxxxxx>
• Subject: [thredds] thredds.filesystem.ControllerOS massive context switching
• From: Brian McKenna <BMcKenna@xxxxxxxxxxxxxx>
• Date: Wed, 31 Jul 2013 11:45:50 -0400

I'm having some issues with THREDDS and a very high number of 
interrupts/context switches killing the server's performance. Increasing the 
LOG level shows the high switching related to thredds.filesystem.ControllerOS. 
As soon as the messages pasted below begin, the interrupt/context switch number 
jumps from <100 to nearly 20,000. CPU and memory do not appear to be a limiting 
factor, their loads remain low. The filesystem is mounted via NFS, but 
read/write speeds (via dd) seem fine, greater than 150 megabytes/s. Those 
speeds are of course without the massive context switching overhead.

Despite <update startup ="nocheck" /> being declared (see catalog below), 
ControllerOS seems to be called in a loop somewhere and appears to be listing 
every directory and file within and repeats itself once complete.

Is there a way to avoid such heavy context switching related to this listing of 
files, possibly  by consolidating the files to a single location, or disabling 
the 'listing' being done by ControllerOS all together?

Thanks,
Brian McKenna

threddsServlet.log
-----------------------

2013-07-31T09:06:53.614 -0400 [     11067][        ] DEBUG - 
thredds.filesystem.ControllerOS - List Directory 
/data/maracoos/ensemble/WRF_ensemble_small

2013-07-31T09:06:53.630 -0400 [     11083][        ] DEBUG - 
thredds.filesystem.ControllerOS -   File 
/data/maracoos/ensemble/WRF_ensemble_small/2012.02.10.00

2013-07-31T09:06:53.633 -0400 [     11086][        ] DEBUG - 
thredds.filesystem.ControllerOS - List Directory 
/data/maracoos/ensemble/WRF_ensemble

2013-07-31T09:06:53.634 -0400 [     11087][        ] DEBUG - 
thredds.filesystem.ControllerOS -   File 
/data/maracoos/ensemble/WRF_ensemble_small/2012.06.03.00

2013-07-31T09:06:53.638 -0400 [     11091][        ] DEBUG - 
thredds.filesystem.ControllerOS -   File 
/data/maracoos/ensemble/WRF_ensemble_small/2012.09.30.00

2013-07-31T09:06:53.641 -0400 [     11094][        ] DEBUG - 
thredds.filesystem.ControllerOS -   File 
/data/maracoos/ensemble/WRF_ensemble_small/2012.03.05.00

2013-07-31T09:06:53.645 -0400 [     11098][        ] DEBUG - 
thredds.filesystem.ControllerOS -   File 
/data/maracoos/ensemble/WRF_ensemble_small/2012.03.14.12

2013-07-31T09:06:53.648 -0400 [     11101][        ] DEBUG - 
thredds.filesystem.ControllerOS -   File 
/data/maracoos/ensemble/WRF_ensemble_small/2012.10.05.12

2013-07-31T09:06:53.651 -0400 [     11104][        ] DEBUG - 
thredds.filesystem.ControllerOS -   File 
/data/maracoos/ensemble/WRF_ensemble_small/2012.10.01.00

2013-07-31T09:06:53.653 -0400 [     11106][        ] DEBUG - 
thredds.filesystem.ControllerOS - List Directory /data/maracoos/ensemble/AKQ_WRF

2013-07-31T09:06:53.659 -0400 [     11112][        ] DEBUG - 
thredds.filesystem.ControllerOS -   File 
/data/maracoos/ensemble/WRF_ensemble/2012.02.10.00

2013-07-31T09:06:53.660 -0400 [     11113][        ] DEBUG - 
thredds.filesystem.ControllerOS -   File 
/data/maracoos/ensemble/WRF_ensemble_small/2012.06.01.00

2013-07-31T09:06:53.664 -0400 [     11117][        ] DEBUG - 
thredds.filesystem.ControllerOS -   File 
/data/maracoos/ensemble/WRF_ensemble/2012.06.03.00

2013-07-31T09:06:53.664 -0400 [     11117][        ] DEBUG - 
thredds.filesystem.ControllerOS -   File 
/data/maracoos/ensemble/WRF_ensemble_small/2012.11.16.00

2013-07-31T09:06:53.668 -0400 [     11121][        ] DEBUG - 
thredds.filesystem.ControllerOS -   File 
/data/maracoos/ensemble/WRF_ensemble/2011.12.19.00

2013-07-31T09:06:53.668 -0400 [     11121][        ] DEBUG - 
thredds.filesystem.ControllerOS -   File 
/data/maracoos/ensemble/WRF_ensemble_small/2012.05.04.12


featureCollection from catalog
---------------------
<featureCollection featureType="FMRC" name="Ensemble WRF (large grid) Forecast 
Aggregation Collection" harvest="true" path="WRF-Large/Agg-Collection.nc">
   <metadata inherited="true">
     <serviceName>agg</serviceName>
     <dataFormat>NetCDF</dataFormat>
   </metadata>
   <collection 
spec="/data/maracoos/ensemble/WRF_ensemble/**/#yyyy.MM.dd.HH#.*\.nc$" name=" 
WRF_Large_Ensemble" olderThan="5 min" />
   <update startup="nocheck" trigger="allow" />
   <fmrcConfig regularize="true" datasetTypes="Best Files Runs" />
   <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
     <variable name="lat" orgName="g0_lat_0">
       <attribute name="standard_name" value="latitude" />
     </variable>
     <variable name="lon" orgName="g0_lon_1">
       <attribute name="standard_name" value="longitude" />
     </variable>
     <variable name="u" orgName="U_GRD_GDS0_HTGL_10">
       <attribute name="standard_name" value="eastward_wind_velocity" />
       <attribute name="coordinates" value="time lat lon" />
     </variable>
     <variable name="v" orgName="V_GRD_GDS0_HTGL_10">
       <attribute name="standard_name" value="northward_wind_velocity" />
       <attribute name="coordinates" value="time lat lon" />
     </variable>
     <variable name="pressure" orgName="PRMSL_GDS0_SFC_10">
       <attribute name="coordinates" value="time lat lon" />
     </variable>
     <variable name="temperature" orgName="TMP_GDS0_HTGL_10">
       <attribute name="coordinates" value="time lat lon" />
     </variable>
     <variable name="humidity" orgName="R_H_GDS0_HTGL_10">
       <attribute name="coordinates" value="time lat lon" />
     </variable>
   </netcdf>
 </featureCollection>

• Follow-Ups:
  • Re: [thredds] thredds.filesystem.ControllerOS massive context switching
    • From: John Caron