Re: Thredds out of memory

• To: John Caron <caron@xxxxxxxxxxxxxxxx>
• Subject: Re: Thredds out of memory
• From: Tennessee Leeuwenburg <t.leeuwenburg@xxxxxxxxxx>
• Date: Wed, 06 Apr 2005 15:03:52 +1000

_From owner-netcdf-java@xxxxxxxxxxxxxxxx Tue Apr  5 23:09:54 2005
Received: (from majordo@localhost)
        by unidata.ucar.edu (UCAR/Unidata) id j3657t0r012120
        for netcdf-java-out; Tue, 5 Apr 2005 23:07:55 -0600 (MDT)
Received: from gale.ho.bom.gov.au (gale.ho.bom.gov.au [134.178.6.76])
        by unidata.ucar.edu (UCAR/Unidata) with ESMTP id j3657Vv2012108;
        Tue, 5 Apr 2005 23:07:37 -0600 (MDT)
Organization: UCAR/Unidata
Keywords: 200504060507.j3657Vv2012108
Received: from [134.178.2.87] (kahless.ho.bom.gov.au [134.178.2.87])
        by gale.ho.bom.gov.au (8.9.3 (PHNE_29774)/8.9.3) with ESMTP id FAA17501;
        Wed, 6 Apr 2005 05:07:24 GMT
User-Agent: Mozilla Thunderbird 1.0 (X11/20041206)
X-Accept-Language: en-us, en
MIME-Version: 1.0
CC: support-netcdf-java@xxxxxxxxxxxxxxxx,
        netcdf-java <netcdf-java@xxxxxxxxxxxxxxxx>
References: <42423D15.3080204@xxxxxxxxxx> <4242E000.4020807@xxxxxxxxxxxxxxxx> 
<42489127.3080901@xxxxxxxxxx> <4249A930.5040404@xxxxxxxxxxxxxxxx> 
<424A5BF6.1070807@xxxxxxxxxx> <424ADF41.4020006@xxxxxxxxxxxxxxxx> 
<424B866C.8070506@xxxxxxxxxx> <424C1637.6090800@xxxxxxxxxxxxxxxx>
X-Enigmail-Version: 0.90.0.0
X-Enigmail-Supports: pgp-inline, pgp-mime
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Sender: owner-netcdf-java@xxxxxxxxxxxxxxxx
Precedence: bulk

For anyone who's interested, I am now quite happily serving large data 
sets via HTTP to thredds. The key was to re-write my java servlet in 
python. I think object creation overhead was killing me because of the 
sheer number of independant requests made by thredds. I used a default 
window size of 65,536 bytes, and was able to get 1.1Mb/s throughput 
downloading from thredds. My test file was 590Mb.

I like this approach, because my custom data source is neatly de-coupled 
from thredds code. All I have to do is deal with the HTTP protocol, and 
everything is handled for me.

If anyone else is having difficulty integrating a new data source into 
thredds, feel free to contact me for my code.

Thanks for everyone's help...

Cheers,
-Tennessee

• Follow-Ups:
  • Re: Thredds out of memory
    • From: John Caron

• References:
  • Re: Thredds out of memory
    • From: John Caron