Re: [thredds] How to download bulk datasets?

• To: Heiko Klein <Heiko.Klein@xxxxxx>
• Subject: Re: [thredds] How to download bulk datasets?
• From: Don Murray <dmurray@xxxxxxxxxxxxxxxx>
• Date: Tue, 04 May 2010 07:04:54 -0600

Heiko-

Don Murray

Heiko Klein wrote:

Hi,

we are moving more and more from our ftp-solutions to thredds with http
and opendap enabled.

Some users complain about this solution, since it is no longer possible
to download bulk datasets, that is, all files in one directory. Our
ftp-server supported 'ls' and several ftp-clients have support for that
so e.g.
ftp ftp.my.server
$ cd directory
$ mget *.nc
worked well.

There are some http-downloader which support mirroring of a directory
which would be comparable, but this requires a proper directory-listing
for the http-download.

An example:
http://thredds.met.no/thredds/catalog/data/met.no/ice-drift/
contains daily files of several years. To clicks further
http://thredds.met.no/thredds/fileServer/data/met.no/ice-drift/ice-drift_ice_drift_nh_polstere-625_multi-oi_200912311200-201001021200.nc
is one of those files.

wget -r -l1 --no-parent -A.nc
'http://thredds.met.no/thredds/fileServer/data/met.no/ice-drift/'
was my best try to get all netcdf-files in the ice-drift catalog.
Unfortunately, this requires a ice-drift/index.html (or
directory-listing) which doesn't exists.


Does anybody knows about a solution to download several (hundred) files
from a thredds-server in a simple way?
I even thought about aggregation, but as far as I see, this doesn't work
with the http-downloader, but requires a opendap client (i.e. nco),
which might be to complicated, and might lead to errors if products
change of the years (better resolution, updated metadata...)

Best regards,

Heiko

_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx

For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/

--
*************************************************************
Don Murray                               UCAR Unidata Program
dmurray@xxxxxxxxxxxxxxxx                        P.O. Box 3000
(303) 497-8628                              Boulder, CO 80307
http://www.unidata.ucar.edu/staff/donm
*************************************************************

• Follow-Ups:
  • Re: [thredds] How to download bulk datasets?
    • From: jeff mcwhirter

• References:
  • [thredds] How to download bulk datasets?
    • From: Heiko Klein