RE; Apparent Latency Problem Solution

• To: ldm-users@xxxxxxxxxxxxxxxx
• Subject: RE; Apparent Latency Problem Solution
• From: Jim Koermer <koermer@xxxxxxxxxxxxxxxxx>
• Date: Fri, 01 Dec 2000 10:34:53 -0500

Hello All,

Last week, we had a good discussion on the latency problems that our
school (Plymouth State) and others were experiencing over the FOS feed
because of the injection of NOAAPORT NWSTG data into the stream. It
seemed like more often than not that we were experiencing latencies > 60
minutes and we were losing a great deal of data or getting it too late.

Russ said a few things in his e-mail that got me thinking and
experimenting, especially relating to the fact that some machines may
not be able of handling all the rpc calls. In fact, our primary IDD
machine was an 1996 vintage RS-6000 with ~67MHz processor and 256MB of
RAM. It had been working well until the NOAAPORT (including ETA) data
expansion.

As a result, I had my feed sites, allow a much faster Dell PowerEdge
server with dual PII-MHz processors and 1 GB RAM, to request data. I
kept the IDS|DDPLUS feeds on the RS6000 and have MCIDAS and HDS data
going into the Dell machine. 

Since doing this, my FOS latencies for DDPLUS|IDS are down to usually
less than 2 minutes. I don't know the latency stats for the Dell, yet,
since the stats perl script uses what I think is a non-ASCII option with
"ls" (i.e. -x - we are working on an alternative). However, for about
the first time in recent memory, we are getting complete ETA grib files
(> 200mb) in very timely fashion, so I think the latencies are probably
quite low on the Dell. This performance has continued for several days,
since making the transition.

The lesson seems to be that the pipe may be fine, but the ingestor may
need to be upgraded to greatly improve IDD data reception and
performance.

BTW, this came together at just the right time, since our NOAAPORT NWSTG
receiver has gone south over the last few days and needs some work.

                                Jim
-- 
James P. Koermer             E-Mail: koermer@xxxxxxxxxxxxxxxxx
Professor of Meteorology     Office Phone: (603)535-2574
Natural Science Department   Office Fax: (603)535-2723
Plymouth State College       WWW: http://vortex.plymouth.edu/
Plymouth, NH 03264