Re: 20020401: LDM performance vs scp?

• To: Joe Van Andel <vanandel@xxxxxxxxxxxx>
• Subject: Re: 20020401: LDM performance vs scp?
• From: Anne Wilson <anne@xxxxxxxxxxxxxxxx>
• Date: Mon, 01 Apr 2002 14:34:06 -0700

Hi Joe,

As others have pointed out, the LDM provides features over and above ftp
and scp such as duplicate detection, acting on data products, time
constraints, etc.  I wish to add the point that the LDM was designed so
that there was no central data source, and a network based on it would
be scalable, e.g., adding sites doesn't significantly degrade
performance.  So while using ftp or scp might be the best route for you,
we don't currently have the infrastructure to support everyone's use of
that approach.    

With current technology and with 130+ institutions (over 220 hosts)
participating in the IDD, using ftp, http, and scp on a pull basis would
be difficult.  If everyone knew for sure when a product would be at a
site, and everyone set up cron jobs that never failed, then we could
keep the hierarchical structure and the whole IDD could go to some pull
based approach.  But I don't think that's feasible. Or, if we had one
super site that could handle 130+ sites trying to constantly pull data
from it then everyone could use their favorite copy mechanism, but
that's not currently feasible either.

Having said that, it could be the case that sftp, http, or scp is faster
than the LDM's RPC approach, especially on a single product basis.  I
really don't know.  I suppose the LDM could be coded to ftp files to
other sites on a push/subscription basis like it does now.  That does,
however, raise design questions, e.g., use the ftp port *and* port 388? 
Both ftp and http involve multiple connections, whereas the LDM uses one
persistent connection, supposedly saving overhead.

Anne
-- 
***************************************************
Anne Wilson                     UCAR Unidata Program            
anne@xxxxxxxxxxxxxxxx                  P.O. Box 3000
                                  Boulder, CO  80307
----------------------------------------------------
Unidata WWW server       http://www.unidata.ucar.edu/
****************************************************



> >
> > ------- Forwarded Message
> >
> > Date:    Fri, 29 Mar 2002 14:49:24 -0700
> > From:    Joe Van Andel <vanandel@xxxxxxxxxxxx>
> > To:      ldm-users@xxxxxxxxxxxxxxxx
> > cc:      anne@xxxxxxxxxxxxxxxx, Russ Rew <russ@xxxxxxxxxxxxxxxx>
> > Subject: LDM performance vs scp?
> >
> > This morning, I was benchmarking file transfers using LDM vs copying the
> > same file with 'scp'.  On the two files I checked, (1237KB and
> > 2446KB),LDM took 3 times as long to send the file compared to scp.
> >
> > Since I'm already short of bandwidth on my T-1 line, I'm quite concerned
> > that I can not afford to use LDM.
> >
> > Has anyone else benchmarked LDM to determine how fast it copies files,
> > vs alternatives (ftp, http, scp)?
> >
> > Any advice on how to improve LDM performance?
> >
> > --
> > Joe VanAndel
> > National Center for Atmospheric Research
> > http://www.atd.ucar.edu/~vanandel/
> > Internet: vanandel@xxxxxxxx
> >

• References:
  • 20020401: LDM performance vs scp?
    • From: Steve Chiswell