Re: FW: Catalog generator crawling remote OPenDAP/DODS servers

• To: Bas Retsios <retsios@xxxxxx>
• Subject: Re: FW: Catalog generator crawling remote OPenDAP/DODS servers
• From: Ethan Davis <edavis@xxxxxxxxxxxxxxxx>
• Date: Fri, 16 Jun 2006 16:13:30 -0600

Hi Bas,

I hope this is helpful. Sorry the existing code isn't really working.

Let me know if I can be of further assistance.

Ethan

Bas Retsios wrote:

Hello Ethan,

Valentijn Venus asked me to assist him in getting the remote DODS catalog generator running. He forwarded me your recent email communication (see below).

We have spent alot of time trying to accomplish this with the cataloggen servlet, in Thredds 3.10. As far as I understood it is a matter of adding a correct XML file as a new task with http://<servername>:8080/thredds/cataloggen/admin/ , and have "Period(minutes)" > 0.

Unfortunately I do not have a proper working example of an XML file that crawls a remote DODS server. The existing example (under config/examples/cetGenConf.exampleDods.xml) does nothing, perhaps because the links in that file are outdated. "Does nothing" means that after the "Initial Delay", the "Resulting Catalog" is an XML file of 9 lines, which (to my opinion) does not seem to contain any useful information.

Could you please send me a working version of config/examples/cetGenConf.exampleDods.xml that I could try? (I'll also be happy with another example, as long it is about remode DODS data). Or if you suspect there is another problem in our approach, give another hint?

Our intention was to follow your suggestion and transform the DodsDirDataSource into a CrawlableDataset, but first I would like to understand how the original "cataloggen servlet" code works.

Best regards,

Bas

--
Ir. V. (Bas) Retsios
Software Developer
IT Department, Sector Remote Sensing & GIS
International Institute for Geo-information Science and Earth Observation (ITC)
P.O. Box 6,  7500 AA Enschede, The Netherlands
Phone +31 (0)53 4874 573, telefax +31 (0)53 4874 335
E-mail retsios@xxxxxx, Internet http://www.itc.nl

Valentijn Venus wrote:

-----Original Message-----

From: Ethan Davis [mailto:edavis@xxxxxxxxxxxxxxxx] Sent: Wednesday, June 07, 2006 12:29 AM

To: Valentijn Venus
Cc: support-thredds@xxxxxxxxxxxxxxxx
Subject: Re: Catalog generator crawling remote OPenDAP/DODS servers

Hi Valentijn,

Actually, the CatalogGen servlet is included as part of the TDS. If you
are writing configuration files that include <datasetScan> elements, you
are using the newer TDS configuration. The URLs for this stuff are of
this form: "http://<server>:<port>/thredds/...". If you are writing
configuration files that include <catalogGenConfig> elements, you are
using the old catGen framework. The CatalogGen servlet is at
"http://<server>:<port>/thredds/cataloggen/...".

The 3.8 code (and now the 3.10 code) are available at

ftp://ftp.unidata.ucar.edu/pub/thredds/tmp/

The code for both the old and new style catalog generation are included.

The old implementation for crawling OPeNDAP/DODS servers is in
src/thredds/cataloggen/config/DodsDirDatasetSource.java. In the new
framework, an implementation of
src/thredds/crawlabledataset/CrawlableDataset.java would have to be
written to handle crawling an OPeNDAP/DODS server. The old stuff might
be easy to get up and running quickly but since the plan is to move
everything to the CrawlableDataset framework, I would encourage you to
go that route.

Let me know what you decide.

Ethan

Valentijn Venus wrote:

Thanks for you response. I guess with thredds.war 3.8 i'm using the

newer TDS configuration which also generates catalogs?

Can you send me a pointer to that old code please? Thanks again...

Valentijn

-----Original Message-----
From: Ethan Davis [mailto:edavis@xxxxxxxxxxxxxxxx]
Sent: Mon 6/5/2006 19:58
To: Valentijn Venus
Cc: support-thredds@xxxxxxxxxxxxxxxx
Subject: Re: Catalog generator crawling remote OPenDAP/DODS servers

Hi Valentijn,

Valentijn Venus wrote:

Hi Ethan,

First a fundamental questions related to THREDDS: the idea for the catalog generator is to crawl whatever source of data you have (local

or
remote) and generate a catalog, correct?

That is the idea but unfortunately crawling a local file system is all

that is currently implemented.

Now I'm looking for some

directions on how to explore all sub-directories on a remote DODS server. The following servers I have in mind:

MODIS AQUA/TERRA:
http://g0dup05u.ecs.nasa.gov/OPENDAP_DP/long_term/MOAA/MYD04_L2.004/2
002
.07.16/
TOMS Ozone:
http://reason.gsfc.nasa.gov/opendap-bin/nph-dods/FTP_DATA/Giovanni/OP
S/T
OMS/EP/

Recently they moved these servers, and since then only the DODS Dataset Access Form for TOMS seems to work. When accessing one of these HDF files (i.e.

http://reason.gsfc.nasa.gov/opendap-bin/nph-dods/FTP_DATA/Giovanni/OP
S/T

OMS/EP/TOMS-EP_L3-TOMSEPL3_1996m0725_v8.HDF.html) and once the form is setup, I can see the resulting "get"/"post" parameters trailing

the "?"

in a DODS url (...?Ozone[0:1:179][0:1:287]) How does this work once the catalog is generated, and a client (IDV, Matlab, etc.) accesses it in real-time? I guess THREDDS supports geographic/paramter subsetting in a similair manner, but how does it work technically?

Generally, the THREDDS catalog provides the base DODS URL and the client needs to understand the OPeNDAP/DODS protocol (the ".dds", ".das", ".dods" extensions and all the stuff after the "?"). THREDDS allows for metadata to indicate the geographic coverage of a dataset but it doesn't support subsetting a particular dataset. The actual subsetting all dependent on the protocol being used to access the dataset, in this case OPeNDAP.

Back to the catalog generation. At one point I did have OPeNDAP server

crawling capabilities in the catalog generator but no one was using it

and I don't think it is still working after a few major modifications.

Are you using the stand-alone CatalogGen application, the catalog generation servlet interface, or the newer TDS configuration which also generates catalogs? The TDS config is a completely different framework than that used by the app and servlet. The code (non-TDS framework) for crawling OPeNDAP servers is still around if you want to

look into getting it running again. However, I'm planning on moving the CatGen app and servlet over to the new TDS framework at some point

but that may be awhile.

Hope this helps.

Ethan

Cheers, Valentijn

whom they are addressed.

--
Ethan R. Davis                                Telephone: (303) 497-8155
Software Engineer                             Fax:       (303) 497-8690
UCAR Unidata Program Center                   E-mail:    edavis@xxxxxxxx
P.O. Box 3000
Boulder, CO  80307-3000                       http://www.unidata.ucar.edu/
---------------------------------------------------------------------------