Re: Thredds out of memory

• To: John Caron <caron@xxxxxxxxxxxxxxxx>
• Subject: Re: Thredds out of memory
• From: Tennessee Leeuwenburg <t.leeuwenburg@xxxxxxxxxx>
• Date: Mon, 11 Apr 2005 10:39:13 +1000

_From owner-netcdf-java@xxxxxxxxxxxxxxxx Sun Apr 10 18:45:23 2005
Received: (from majordo@localhost)
        by unidata.ucar.edu (UCAR/Unidata) id j3B0hNDY002037
        for netcdf-java-out; Sun, 10 Apr 2005 18:43:23 -0600 (MDT)
Received: from gale.ho.bom.gov.au (gale.ho.bom.gov.au [134.178.6.76])
        by unidata.ucar.edu (UCAR/Unidata) with ESMTP id j3B0h5v2002026;
        Sun, 10 Apr 2005 18:43:08 -0600 (MDT)
Organization: UCAR/Unidata
Keywords: 200504110043.j3B0h5v2002026
Received: from [134.178.2.87] (kahless.ho.bom.gov.au [134.178.2.87])
        by gale.ho.bom.gov.au (8.9.3 (PHNE_29774)/8.9.3) with ESMTP id AAA07448;
        Mon, 11 Apr 2005 00:42:58 GMT
User-Agent: Mozilla Thunderbird 1.0 (X11/20041206)
X-Accept-Language: en-us, en
MIME-Version: 1.0
CC: support-netcdf-java@xxxxxxxxxxxxxxxx,
        netcdf-java <netcdf-java@xxxxxxxxxxxxxxxx>
References: <42423D15.3080204@xxxxxxxxxx> <4242E000.4020807@xxxxxxxxxxxxxxxx> 
<42489127.3080901@xxxxxxxxxx> <4249A930.5040404@xxxxxxxxxxxxxxxx> 
<424A5BF6.1070807@xxxxxxxxxx> <424ADF41.4020006@xxxxxxxxxxxxxxxx> 
<424B866C.8070506@xxxxxxxxxx> <424C1637.6090800@xxxxxxxxxxxxxxxx> 
<42536DB8.1090905@xxxxxxxxxx> <4254001C.6010708@xxxxxxxxxxxxxxxx>
X-Enigmail-Version: 0.90.0.0
X-Enigmail-Supports: pgp-inline, pgp-mime
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Sender: owner-netcdf-java@xxxxxxxxxxxxxxxx
Precedence: bulk

John Caron wrote:

> Tennessee Leeuwenburg wrote:
>
>> For anyone who's interested, I am now quite happily serving large 
>> data sets via HTTP to thredds. The key was to re-write my java 
>> servlet in python. I think object creation overhead was killing me 
>> because of the sheer number of independant requests made by thredds. 
>> I used a default window size of 65,536 bytes, and was able to get 
>> 1.1Mb/s throughput downloading from thredds. My test file was 590Mb.
>>
>> I like this approach, because my custom data source is neatly 
>> de-coupled from thredds code. All I have to do is deal with the HTTP 
>> protocol, and everything is handled for me.
>>
>> If anyone else is having difficulty integrating a new data source 
>> into thredds, feel free to contact me for my code.
>>
>> Thanks for everyone's help...
>>
>> Cheers,
>> -Tennessee
>
>
> Are you sure the key was moving to python, and not moving to a 65K 
> buffer?


That could well be the case, as I did both at once. Java is a lovely 
language, however I can also see how it could suffer in the face of 
repeated object creation when handling many small requests. In this 
case, however, the baby and the bathwater can't be easily differentiated :)

I'm going to stick with the Python code on the grounds that it works, 
and I've got more urgent things to do right now, but it could be that 
Java could do the same job.

Cheers,
-T

• References:
  • Re: Thredds out of memory
    • From: John Caron