[ldm-users] 20100429: Feedtype confusion (cont.)

• To: Peter Laws <plaws@xxxxxx>
• Subject: [ldm-users] 20100429: Feedtype confusion (cont.)
• From: Tom Yoksas <yoksas@xxxxxxxxxxxxxxxx>
• Date: Thu, 29 Apr 2010 17:36:27 -0600

Hi Peter,

>On 04/29/10 13:26, Steve Emmerson wrote:
>> On 04/29/2010 11:20 AM, daryl herzmann wrote:
>>> In that case, you really should use slightly different feed requests to
>>> keep LDM from bouncing back and forth.  For example
>>>
>>> REQUEST EXP upstream ".*"
>>> REQUEST EXP upstream2 "(.*)"
>>
>> It's your call, but you should know that doing this will double the
>> bandwidth usage.

re:
>Not clear to me which end this would be on.  Would my downstream clients do 
>that?

Yes.

re:
>They are - that's why we have two servers.  They point at both 
>ingest systems for redundancy.

Very good.

Comment:

Downstreams initiate feed requests via REQUEST lines in their
~ldm/etc/ldmd.conf files.  Making the exact same request to more than one
upstream that should have the exact same set of products is the easiest
way to provide for backup feeds.  What Daryl was suggesting was to specify
different, but functionally equivalent, feed requests so that the LDM's
"ping-poing" failover strategy is avoided.  This has the advantage of
reducing the number of log messages in your ~ldm/logs/ldmd.log file, but
it has the disadvantage of using twice the network bandwidth. Duplicate
products will be detected and rejected at the point when an attempt is
made to write a product to the LDM queue; if the product already was
received from the "other" feed, the current product will be discarded.

re:
>If I'm to do that, presumably for redundancy, what would be the 2nd system 
>upstream?

Are you referring to providing redundancy on your ingest machines?
You already are providing redundancy on your downstream machines.

re:
>Currently, we pull most stuff from idd.unidata.ucar.edu  (and 
>some other products from other servers).  Should I add a 2nd one?

Our recommendation is that a site has a primary machine that requests
data from one or more upsteam nodes in the IDD.  That machine should
then relay data to however many internal machines are needed.

re:
>If so, who?

There are a variety of other sites that can provide one or more
datastreams.  Which one(s) you use depends on which datastreams you are
requesting.  For instance, if you are receiving CONDUIT, you may want to
have one request to idd.unidata.ucar.edu and a second to idd.meteo.psu.edu
(Penn State).  If you want datastreams that originate in NOAAPort,
you could get a redundant feed at a variety of upstream hosts (e.g.,
unidata2.ssec.wisc.edu, idd.cise-nsf.gov, etc.).  Again, it all depends
on which datastreams you want/need.

Cheers,

Tom
--
+----------------------------------------------------------------------------+
* Tom Yoksas                                            UCAR Unidata Program *
* (303) 497-8642 (last resort)                                 P.O. Box 3000 *
* yoksas@xxxxxxxxxxxxxxxx                                  Boulder, CO 80307 *
* Unidata WWW Service                            http://www.unidata.ucar.edu/*
+----------------------------------------------------------------------------+

• References:
  • Re: [ldm-users] Feedtype confusion
    • From: Peter Laws