Re: Nomads Forecast Aggregation

• To: "Steven.Anthony" <Steven.Anthony@xxxxxxxx>
• Subject: Re: Nomads Forecast Aggregation
• From: John Caron <caron@xxxxxxxxxxxxxxxx>
• Date: Thu, 14 Jun 2007 11:27:52 -0600

Hi Steven:

try adding inherited="true":

<datasetFmrc name="gfs-test-FMRC" collectionType="ForecastModelRuns" harvest="true" 
path="fmrc/gfs_3">

       <metadata inherited="true">
               <dataFormat>GRIB-1</dataFormat>
               <serviceName>fmrcServices</serviceName>
       </metadata>


Steven.Anthony wrote:

John and Ethan,
The Nomads team here at NCDC has been looking into the FMRC capability
of the TDS for a while.  I have tried many times with some success, but
our main datasets still refuse to cooperate.  I feel that I am doing
something wrong, but can't seem to fix it.  Can you take a look at our
catalogs/data to see if there is something that I am doing wrong?

The Data:
For my test, I have decided to use our GFS data, and then work from
there if it works.
The data is standard GFS one degree data from NCEP.  It is in GRIB-1
format.  Each model run contains many files (one forecast hour per file)
and we store the 4 runs from each day in a common folder.
The DatasetScan has worked fantastically on this dataset, and is already
in use.

The data is named by our own Nomads conventions:
gfs_3_yyyymmdd_hhhh_fff.grb
where:
gfs_3 = model name & number
yyyy = year
mm = month
dd = day
hhhh = model run time (0000, 0600, 1200, 1800)
fff = forecast hour

You can view the data and its structure at our public Nomads TDS. http://nomads-d.ncdc.noaa.gov:8081/thredds/catalog.html

in the GFS directory.


The Catalog:
First, I just updated our development server to the latest verstion
(3.16.10 Build Date = 2007-06-14 00:14:30)
Our DatasetScan catalogs still work, but our FMRC catalogs still only
half work.

here is the truncated version of our catalog.xml file (working
DatasetScans, comments, etc. removed):

<?xml version="1.0" encoding="UTF-8"?>
<catalog
    xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0";
    xmlns:xlink="http://www.w3.org/1999/xlink";
    name="NCDC-NOMADS THREDDS Data Server II"
    version="1.0.2">

<!-- Services -->
<service name="fmrcServices" serviceType="Compound" base="">
  <service name="ncdods" serviceType="OpenDAP" base="/thredds/dodsC/"/>
  <service name="wcs" serviceType="WCS" base="/thredds/wcs/"/>
  <service name="ncServer" serviceType="NetcdfServer"
base="/thredds/ncServer/"/>
</service>


<datasetFmrc name="gfs-test-FMRC" collectionType="ForecastModelRuns"
harvest="true" path="fmrc/gfs_3">
        <metadata>
                <dataFormat>GRIB-1</dataFormat>
                <serviceName>fmrcServices</serviceName>
        </metadata>



        <netcdf
xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";
enhance="true">
                <aggregation dimName="run"
type="forecastModelRunSingleCollection"
fmrcDefinition="NCEP-GFS-Global_onedeg.fmrcDefinition.xml"
timeUnitsChange="true" >

                        <scanFmrc
location="/nomads3_data/raid2/noaaport/merged/gfs-hi/200705/20070531"
                        regExp=".*\.grb$"
                        runDateMatcher="#gfs_3_#yyyyMMdd_HH"
                        forecastOffsetMatcher="HHH#.grb#"/>

                </aggregation>
        </netcdf>

        <fmrcInventory
location="/nomads3_data/raid2/noaaport/merged/gfs-hi/200705/20070531"
suffix=".grb" fmrcDefinition="NCEP-GFS-Global_onedeg.fmrcDefinition.xml" />
</datasetFmrc>

</catalog>



This catalog works as in I don't receive an error in the web browser,
however the models.log file has this this exception.
[14/Jun/2007:10:58:11] ERROR thredds.catalog.InvDatasetFmrc : Error
making catalog for fmrc/gfs_3
java.lang.IllegalArgumentException: The service must not be null.
        at
thredds.cataloggen.CollectionLevelScanner.<init>(CollectionLevelScanner.java:237)
        at
thredds.cataloggen.StandardCatalogBuilder.setupAndScan(StandardCatalogBuilder.java:212)
        at
thredds.cataloggen.StandardCatalogBuilder.generateCatalog(StandardCatalogBuilder.java:133)
        at
thredds.cataloggen.DatasetScanCatalogBuilder.generateCatalog(DatasetScanCatalogBuilder.java:119)
        at
thredds.catalog.InvDatasetScan.makeCatalogForDirectory(InvDatasetScan.java:504)
        at
thredds.catalog.InvDatasetFmrc.makeCatalogScan(InvDatasetFmrc.java:445)
        at
thredds.catalog.InvDatasetFmrc.makeCatalog(InvDatasetFmrc.java:158)
        at
thredds.servlet.DataRootHandler.makeDynamicCatalog(DataRootHandler.java:1167)
        at
thredds.servlet.DataRootHandler.getCatalog(DataRootHandler.java:1135)
        at
thredds.servlet.DataRootHandler.processReqForCatalog(DataRootHandler.java:1050)
        at thredds.servlet.CatalogServlet.doGet(CatalogServlet.java:78)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
        at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
        at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
        at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
        at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
        at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
        at java.lang.Thread.run(Thread.java:595)

This sounds about right as the FMRC is parsing the dates right, and
generating the catalog, however, it does not provide the access links
that should be there (OpenDAP, WCS and NetCDF Server)

If you want any more information, feel free to ask.
I appreciate your level of support and commitment, and we look forward
to getting the FMRC aggregation working for all of our datasets so we
can start to use it more.

Thanks again
 - Steven