NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

[conduit] 20210830: Re: High CONDUIT latencies from vm-lnx-conduit2.ncep.noaa.gov

Hi Jesse,

On 8/30/21 5:16 PM, Jesse Marks - NOAA Affiliate wrote:
Quick question:  how are you computing these latencies? 

Latency in the LDM/IDD context is the time difference between when a
product is first put into an LDM queue for redistribution and the time
it is received by a downstream machine.  This measure of latency, of
course, requires that the clocks on the originating and receiving
machines be maintained accurately.

re:
More specifically, how do you determine which conduit machine the data is coming from? 

The machine on which the product is inserted into the LDM queue is
available in the LDM transaction.  We provide an website where users
can create graphs of things like feed latencies:

Unidata HomePage
https://www.unidata.ucar.edu

  IDD Operational Status
  https://rtstats.unidata.ucar.edu/rtstats/

    Real-time IDD Statistics -> Statistics by Host
    https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex

The variety of measures of feed quality for the Unidata machine that
is REQUESTing the CNODUIT feed from the NCEP cluster can be found in:

https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex?conduit.unidata.ucar.edu

The latencies being reported by the Unidata machine that is being fed
from the NCEP cluster is:

CONDUIT latencies on conduit.conduit.unidata.ucar.edu:

https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+conduit.unidata.ucar.edu

As you can see, the traces are color color coded, and the label at the
top identifies the source machines for products.

re:
The reason I ask is because I am not seeing any sends of product from conduit2 in the last several days of logs both to our local conduit1 machine and to any distant end users. 

Hmm...  we are.

re:
Also, we have isolated what is likely the issue and will have our team take a closer look in the morning.  I'm hopeful they'll be able to resolve this soon. 

Excellent!  We are hopeful that the source of the high latencies will
be identified and fixed.

Cheers,

Tom
On Mon, Aug 30, 2021 at 5:24 PM Anne Myckow - NOAA Federal <anne.myckow@xxxxxxxx <mailto:anne.myckow@xxxxxxxx>> wrote: 

    Pete,

    Random aside, can you please update your doco to say that
    Dataflow's email list is now nco.dataflow@xxxxxxxx
    <mailto:nco.dataflow@xxxxxxxx> ? I'm CC'ing it here. That other
    email address is going to get turned off within the next year.

    Thanks,
    Anne

    On Wed, Aug 18, 2021 at 4:02 PM Pete Pokrandt <poker@xxxxxxxxxxxx
    <mailto:poker@xxxxxxxxxxxx>> wrote:

        Dear Anne, Dustin and all,

        Recently we have noticed fairly high latencies on the CONDUIT
        ldm data feed originating from the machine
        vm-lnx-conduit2.ncep.noaa.gov
        <http://vm-lnx-conduit2.ncep.noaa.gov>. The feed originating
        from vm-lnx-conduit1.ncep.noaa.gov
        <http://vm-lnx-conduit1.ncep.noaa.gov> does not have the high
        latencies. Unidata and other top level feeds are seeing similar
        high latencies from vm-lnx-conduit2.ncep.noaa.gov
        <http://vm-lnx-conduit2.ncep.noaa.gov>.

        Here are some graphs showing the latencies that I'm seeing:

         From
        
https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd-agg.aos.wisc.edu
        
<https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd-agg.aos.wisc.edu> -
        latencies for CONDUIT data arriving at our UW-Madison AOS ingest
        machine



         From
        
https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex?conduit.unidata.ucar.edu
        
<https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex?conduit.unidata.ucar.edu> (latencies
        at Unidata)



        At least here at UW-Madison, these latencies are causing us to
        lose some data during the large GFS/GEFS periods.

        Any idea what might be causing this?

        Pete




        
<http://www.weather.com/tv/shows/wx-geeks/video/the-incredible-shrinking-cold-pool>-----
        Pete Pokrandt - Systems Programmer
        UW-Madison Dept of Atmospheric and Oceanic Sciences
        608-262-3086  - poker@xxxxxxxxxxxx <mailto:poker@xxxxxxxxxxxx>
-- Anne Myckow 
    Dataflow Team Lead
    NWS/NCEP/NCO



--
Jesse Marks
Dataflow Analyst
NCEP Central Operations
678-896-9420


--
+----------------------------------------------------------------------+
* Tom Yoksas                                      UCAR Unidata Program *
* (303) 497-8642 (last resort)                           P.O. Box 3000 *
* yoksas@xxxxxxxx                                    Boulder, CO 80307 *
* Unidata WWW Service                     http://www.unidata.ucar.edu/ *
+----------------------------------------------------------------------+
  • 2021 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the conduit archives: