Re: [thredds] How to download bulk datasets?

• To: thredds@xxxxxxxxxxxxxxxx
• Subject: Re: [thredds] How to download bulk datasets?
• From: Ethan Davis <edavis@xxxxxxxxxxxxxxxx>
• Date: Mon, 26 Jul 2010 17:12:10 -0600

Hi Jerry,

Thanks for the suggestion. We've heard this type of request a few times.
I have added it to our enhancement wish list so we can try to fit it in
to our plans.

Thanks,

Ethan

On 7/1/2010 1:36 PM, Pan, Jerry Yun wrote:
> I too wish there is a simple link on the server side to allow
> our users to use wget for downloading all files of a collection.
> 
> I think it might be the case that every user would want to do a
> bulk file download from a thredds server, and I think it might
> be the case that a provider should have the burden to have a
> simple link for bulk download all granules.
> 
> We can do some configurations or add a servlet for this, like
> Heiko has done, although I think it would be a nice to have
> feature directly from TDS software. It would seem to me that
> this is can be implemented as a dynamic URL at collection
> level to return a list of HTTP download URLs of the files
> ('fielServer') for the simplest case.
> 
> Comments?
> 
> Thanks,
> -Jerry
> 
>> -----Original Message-----
>> From: thredds-bounces@xxxxxxxxxxxxxxxx 
>> [mailto:thredds-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Heiko Klein
>> Sent: Monday, May 10, 2010 5:18 AM
>> To: John Caron
>> Cc: thredds@xxxxxxxxxxxxxxxx
>> Subject: Re: [thredds] How to download bulk datasets?
>>
>> Hi John,
>>
>> I played a bit more with the catalog.xml. This works well 
>> with wget. I managed now to download all the netcdf-files 
>> from a directory:
>>
>> wget -nc -r -l2 -A.nc   -I /thredds/fileServer/,/thredds/catalog/
>> 'http://dev-vm188/thredds/catalog/osisaf/met.no/ice/'
>>
>> I use here the existing datasetScan catalog.xml file, and 
>> fetch all nc-files up to two links away. Beside the nc-file, 
>> I get the catalog-file of the nc-file (e.g.
>> http://dev-vm188/thredds/catalog/osisaf/met.no/ice/catalog.htm
>> l?dataset=met.no/ice/ice_conc_nh_200911261200_CF.nc),
>> too.
>>
>> A catalog-file in the fileServer would be saver, since the 
>> 2-levels (parent and child) might include other information, 
>> but at least I can offer our users something already now.
>>
>>
>> Best regards,
>>
>> Heiko
>>
>> On 2010-05-06 21:31, John Caron wrote:
>>> Hi Heiko:
>>>
>>> We use catalog.xml exactly because theres no standard html 
>> index format.
>>> A simple java GUI app could make this easy to do, but Im 
>> not clear if 
>>> that would help your case.
>>>
>>> John
[snip]

• References:
  • [thredds] How to download bulk datasets?
    • From: Heiko Klein
  • Re: [thredds] How to download bulk datasets?
    • From: John Caron
  • Re: [thredds] How to download bulk datasets?
    • From: Heiko Klein
  • Re: [thredds] How to download bulk datasets?
    • From: John Caron
  • Re: [thredds] How to download bulk datasets?
    • From: Heiko Klein
  • Re: [thredds] How to download bulk datasets?
    • From: Pan, Jerry Yun