NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

Re: Catalog generator crawling remote OPenDAP/DODS servers

  • To: Valentijn Venus <venus@xxxxxx>
  • Subject: Re: Catalog generator crawling remote OPenDAP/DODS servers
  • From: Ethan Davis <edavis@xxxxxxxxxxxxxxxx>
  • Date: Tue, 06 Jun 2006 16:29:16 -0600
Hi Valentijn,
Actually, the CatalogGen servlet is included as part of the TDS. If you are writing configuration files that include <datasetScan> elements, you are using the newer TDS configuration. The URLs for this stuff are of this form: "http://<server>:<port>/thredds/...". If you are writing configuration files that include <catalogGenConfig> elements, you are using the old catGen framework. The CatalogGen servlet is at "http://<server>:<port>/thredds/cataloggen/...". 

The 3.8 code (and now the 3.10 code) are available at

ftp://ftp.unidata.ucar.edu/pub/thredds/tmp/
The code for both the old and new style catalog generation are included. The old implementation for crawling OPeNDAP/DODS servers is in src/thredds/cataloggen/config/DodsDirDatasetSource.java. In the new framework, an implementation of src/thredds/crawlabledataset/CrawlableDataset.java would have to be written to handle crawling an OPeNDAP/DODS server. The old stuff might be easy to get up and running quickly but since the plan is to move everything to the CrawlableDataset framework, I would encourage you to go that route. 

Let me know what you decide.

Ethan

Valentijn Venus wrote:

Thanks for you response. I guess with thredds.war 3.8 i'm using the newer TDS 
configuration which also generates catalogs?

Can you send me a pointer to that old code please? Thanks again...

Valentijn

-----Original Message-----
From: Ethan Davis [mailto:edavis@xxxxxxxxxxxxxxxx]
Sent: Mon 6/5/2006 19:58
To: Valentijn Venus
Cc: support-thredds@xxxxxxxxxxxxxxxx
Subject: Re: Catalog generator crawling remote OPenDAP/DODS servers
Hi Valentijn, 

Valentijn Venus wrote:


Hi Ethan,

First a fundamental questions related to THREDDS: the idea for the
catalog generator is to crawl whatever source of data you have (local or
remote) and generate a catalog, correct?
That is the idea but unfortunately crawling a local file system is all that is currently implemented. 


Now I'm looking for some
directions on how to explore all sub-directories on a remote DODS
server. The following servers I have in mind:

MODIS AQUA/TERRA:
http://g0dup05u.ecs.nasa.gov/OPENDAP_DP/long_term/MOAA/MYD04_L2.004/2002
.07.16/
TOMS Ozone:
http://reason.gsfc.nasa.gov/opendap-bin/nph-dods/FTP_DATA/Giovanni/OPS/T
OMS/EP/ 
Recently they moved these servers, and since then only the DODS Dataset
Access Form for TOMS seems to work. When accessing one of these HDF
files (i.e.
http://reason.gsfc.nasa.gov/opendap-bin/nph-dods/FTP_DATA/Giovanni/OPS/T
OMS/EP/TOMS-EP_L3-TOMSEPL3_1996m0725_v8.HDF.html) and once the form is
setup, I can see the resulting "get"/"post" parameters trailing the "?"
in a DODS url (...?Ozone[0:1:179][0:1:287]) How does this work once the
catalog is generated, and a client (IDV, Matlab, etc.) accesses it in
real-time? I guess THREDDS supports geographic/paramter subsetting in a
similair manner, but how does it work technically?
Generally, the THREDDS catalog provides the base DODS URL and the client needs to understand the OPeNDAP/DODS protocol (the ".dds", ".das", ".dods" extensions and all the stuff after the "?"). THREDDS allows for metadata to indicate the geographic coverage of a dataset but it doesn't support subsetting a particular dataset. The actual subsetting all dependent on the protocol being used to access the dataset, in this case OPeNDAP.
Back to the catalog generation. At one point I did have OPeNDAP server crawling capabilities in the catalog generator but no one was using it and I don't think it is still working after a few major modifications. Are you using the stand-alone CatalogGen application, the catalog generation servlet interface, or the newer TDS configuration which also generates catalogs? The TDS config is a completely different framework than that used by the app and servlet. The code (non-TDS framework) for crawling OPeNDAP servers is still around if you want to look into getting it running again. However, I'm planning on moving the CatGen app and servlet over to the new TDS framework at some point but that may be awhile. 

Hope this helps.

Ethan


Cheers, Valentijn




--
Ethan R. Davis                                Telephone: (303) 497-8155
Software Engineer                             Fax:       (303) 497-8690
UCAR Unidata Program Center                   E-mail:    edavis@xxxxxxxx
P.O. Box 3000
Boulder, CO  80307-3000                       http://www.unidata.ucar.edu/
---------------------------------------------------------------------------
  • 2006 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: