Re: DODS AS

• To: Joe McLean <joe.mclean@xxxxxxxx>
• Subject: Re: DODS AS
• From: John Caron <caron@xxxxxxxxxxxxxxxx>
• Date: Tue, 09 Sep 2003 17:55:24 -0600

Joe McLean wrote:

Hi John,

I have tried to use an NcML file in the latest Agg Server configuration I downloaded last Friday. I repeatedly receive a java error from the server at http://stout.pmel.noaa.gov:8180/dodsC/

This is the pertinent snippet of the error:

java.lang.NullPointerException

at dods.servers.agg.CatalogServlet.addAccess(CatalogServlet.java:611) at dods.servers.agg.CatalogServlet.readCatalog(CatalogServlet.java:257)

        at dods.servers.agg.CatalogServlet.init(CatalogServlet.java:178)
        at javax.servlet.GenericServlet.init(GenericServlet.java:258)

<catalog name="Test NCML Aggregation" version="0.6">
 <dataset name="Top-Level Dataset" dataType="Grid" serviceName="this">
   <service name="this" serviceType="DODS" base=""/>

   <dataset name="NDBC stdmet buoy data">
      <property name="internalService" value="stdmet"/>
      <dataset name="stdmetNcML Agg test" urlPath="TestNcML.xml"/>
   </dataset>
 </dataset>
</catalog>


with this external URL:  http://stout.pmel.noaa.gov:8180/dodsC/TestNcML.xml

and this internal URL: file://WEB-INF/dodsAS/TestNcML.xml

file:/usr/local/www/html/TOMCAT_DEMO/las/las_servlet/jakarta/webapps/dodsC/WEB-INF/dodsAS/TestNcML.xml

and i get the error message:

message = "CatalogServlet ERROR opening NcDataset TestNcML.xml

   <service name="stdmet" serviceType="NetCDF" 
base="file:///path/something/WEB-INF/dodsAS/"/>
   <service name="stdmet" serviceType="NetCDF" 
base="file:/path/something/WEB-INF/dodsAS/"/>

I used your Catalog and NcML aggregation as templates (following this message) and ctreated the following Both are in $tomcat_root/dodsC/WEB-INF/dodsAS/ the web.xml file points to the correct Catalog.xml file. Any ideas? Is it that the NcML netcdf element can't handle a DODS URL? AmI confused about where the aggregation should take place?

--------------------------------------
AggServTestCatalog.xml
---------------------------------------
<!DOCTYPE catalog SYSTEM
"http://www.unidata.ucar.edu/projects/THREDDS/xml/AggServerCatalog.dtd";>
<catalog name="Test NCML Aggregation" version="0.6"
xmlns="http://www.unidata.ucar.edu/thredds";
xmlns:xlink="http://www.w3.org/1999/xlink";>
        <service name="this" serviceType="DODS" base=""/>
        <service name="stdmet" serviceType="NetCDF"
           base="file://WEB-INF/dodsAS/"/ <file://WEB-INF/dodsAS/>>
        <dataset name="NDBC stdmet buoy data">
            <property name="internalService" value="stdmet"/>

<dataset name="stdmet NcML Agg test" urlPath="TestNcML.xml" />

        </dataset>
</catalog>

-
<catalog name="Test NCML Aggregation" version="0.6">
-
<dataset name="Top-Level Dataset" dataType="Grid" serviceName="this">
<service name="this" serviceType="DODS" base=""/>
<service name="stdmet" serviceType="NetCDF" base="file://WEB-INF/dodsAS/"/>
-
<dataset name="NDBC stdmet buoy data">
<property name="internalService" value="stdmet"/>
<dataset name="stdmetNcML Agg test" urlPath="TestNcML.xml"/>
</dataset>
</dataset>
</catalog>

Also tried full path for service:
<service name="stdmet" serviceType="DODS"

base="file:///usr/local/www/html/TOMCAT_DEMO/las/las_servlet/jakarta/webapps/dodsC/WEB-INF/dodsAS/"/ <file:///usr/local/www/html/TOMCAT_DEMO/las/las_servlet/jakarta/webapps/dodsC/WEB-INF/dodsAS/>>

Also tried serviceType="NetCDF"
<service name="stdmet" serviceType="NetCDF"

---------------------------------------
TestNcML.xml
---------------------------------------
<?xml version='1.0' encoding='UTF-8'?>
<netcdf xmlns='http://www.ucar.edu/schemas/netcdf'
    xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'
    xsi:schemaLocation='http://www.ucar.edu/schemas/netcdf
http://www.unidata.ucar.edu/schemas/netcdfDataset2.xsd' >

  <variable name='recnum' type='int' shape='recnum' >
    <attribute name='long_name' type='string' value='time coordinate' />

<attribute name='units' type='string' value='hours since 2001-09-31 00:00:00 UTC' />

    <values start='0' increment='1' />
  </variable>

  <aggregation dimName='recnum' type='joinExisting' />

<netcdf uri='http://dods.ndbc.noaa.gov/cgi-bin/nph-nc/dods/b42001_2001_09_stdmet.nc'/ <http://dods.ndbc.noaa.gov/cgi-bin/nph-nc/dods/b42001_2001_09_stdmet.nc%27/>> <netcdf uri='http://dods.ndbc.noaa.gov/cgi-bin/nph-nc/dods/b42001_2001_10_stdmet.nc' /> <netcdf uri='http://dods.ndbc.noaa.gov/cgi-bin/nph-nc/dods/b42001_2001_11_stdmet.nc'/ <http://dods.ndbc.noaa.gov/cgi-bin/nph-nc/dods/b42001_2001_11_stdmet.nc%27/>> </netcdf>

---------------------------------------

Joe
------------------------------------------------
John Caron wrote:

ok, you need to download a new dodsC.war file, follow instructions from
this page:

http://www.unidata.ucar.edu/projects/THREDDS/tech/aggServer/AggServerInstall.html

here is an example NcML  aggregation that i tested on 3 of your files;
it assumes a uniform 10 minute increment to add a coordinate variable.
it does a join on the recnum dimension on the listed 3 files. it should
work for you, except that obviously you replace with your files, in time
order, and get the starting time right.

<?xml version='1.0' encoding='UTF-8'?>
<netcdf xmlns='http://www.ucar.edu/schemas/netcdf'
    xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'
    xsi:schemaLocation='http://www.ucar.edu/schemas/netcdf
http://www.unidata.ucar.edu/schemas/netcdfDataset2.xsd' >

  <variable name='recnum' type='int' shape='recnum' >
    <attribute name='long_name' type='string' value='time coordinate' />
    <attribute name='units' type='string' value='minutes since 2001-8-31
23:00:00 UTC' />
    <values start='0' increment='10' />
  </variable>

  <aggregation dimName='recnum' type='joinExisting' />

  <netcdf uri='C:/data/buoy/cwind1.nc'/>
  <netcdf uri='C:/data/buoy/cwind2.nc' />
  <netcdf uri='C:/data/buoy/cwind3.nc'/>

</netcdf>

the agg server configuration looks like this:

<!DOCTYPE catalog SYSTEM
"http://www.unidata.ucar.edu/projects/THREDDS/xml/AggServerCatalog.dtd";>
<catalog name="Test NCML Aggregation" version="0.6"
xmlns="http://www.unidata.ucar.edu/thredds";
xmlns:xlink="http://www.w3.org/1999/xlink";>
        <service name="cwind" serviceType="NetCDF"
base="file:///c:/data/buoy/"/ <file:///c:/data/buoy/>>
        <dataset name="cwind buoy data">
            <property name="internalService" value="cwind"/>

<dataset name="cwind NcML Agg test" urlPath="cwindAgg.xml" />

        </dataset>
</catalog>

where the NcML file (as above) lives at file:///c:/data/buoy/cwindAgg.xml

hopefully this will get you started, let me know how it goes.

BTW, this wont scale too huge numbers of netcdf files, but a few dozen
should be ok. i will need to add some file caching to speed it up, but
the above shoul be ok for small sets of files.

• References:
  • Re: DODS AS
    • From: John Caron