NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Best practice in networking calls for two separate power sources. Rarely
will you see a short circuit cause simultaneous outages unless it manages
to kill two separate battery plants. OR, if they are powering from AC, they
are a mission critical asset, so they should have two power feeds, coming
from two different UPS's.
If they lacked diversity in sources, that's a problem and could lead to an
outage if they had a single circuit failure.
Or, they could have done router software or config updates on both routers
simultaneously, while they were swapped out.
The possibilities are finite, really, and I suspect they'll have someone
investigate.
gerry
On Tue, Feb 14, 2017 at 4:36 PM, <admin@xxxxxxxx> wrote:
>
> Ive been there too Kevin, but most interconnects from circuits to routers
> now use fiber optic jumpers in datacenters. For that exact reason. But we
> weren't there, could have been anything.
>
> Ray
>
> On Tuesday, February 14, 2017 5:29pm, "Kevin W. Thomas" <
> kwthomas@xxxxxxxxxxxxxxxxxx> said:
>
> > I'm far from a hardware expert, but I do know electrical shorts can go
> thru
> > cables and take out lots of hardware.
> >
> > Many years ago, I administered a PDP 11/44. The computer room took a
> hit from
> > lightning that fried a lot of computer boards. The surge went down all
> the
> > cable lines and fried a chip an almost all of the VT100's that were
> attached
> > to it. Distance to the computer room didn't matter.
> >
> > This sounds similar.
> >
> > Kevin W. Thomas
> > Center for Analysis and Prediction of Storms
> > University of Oklahoma
> > Norman, Oklahoma
> > Email: kwthomas@xxxxxx
> >
> >>Stonie is right on here. Another thing I find suspicious is that "Two
> routers
> >> failed simultaneously". The odds of that are next to impossible. More
> likely the
> >> routers responded improperly to an event.
> >>
> >>Ray Weber
> >>MA Skywarn
> >>
> >>On Tuesday, February 14, 2017 4:37pm, "Stonie R. Cooper" <
> stonewall@xxxxxxxxxxxxx>
> >> said:
> >>
> >>> I generally keep my comments off of this list, but the official
> >>> statement is not accurate. As you know, Gilbert, WRIP (NOAA Weather
> >>> Radio) was only partially up and that was WFO dependent - if an WFO
> >>> fired up their CRS, they could manually record hourly forecasts and
> >>> observations.
> >>>
> >>> Likewise, checking the tombstones via local forecast on weather.gov
> >>> resulted in "N/A" for most zip codes I tried during the outage.
> >>>
> >>> From an IT side, I would say this particular event should help
> >>> illustrate the foolhardy paradigm of "putting all your eggs in one
> >>> basket." HPC, NCEP, EMC . . . all were hit with this outage. NWWS,
> >>> EMWIN, and SBN were all down. A primary and secondary router paradigm
> >>> in the same physical location may save lots of money, but is an
> >>> incredible liability for data distribution.
> >>>
> >>> I have always advocated that a live secondary should be placed
> >>> geographically distant from the primary uplink and NCF . . . and KC has
> >>> legacy and national networking infrastructure already in place to make
> >>> it a likely candidate. Or Boulder, at ESRL.
> >>>
> >>> Not that my comments will merit any action, but if pointing out what
> >>> seems obvious to me gains traction, that would be great. I would even
> >>> start filling out my TPS cover sheets for all my memos, and coming in
> on
> >>> over the weekend to work. ("Office Space" reference)
> >>>
> >>> Stonie
> >>>
> >>> On 02/14/2017 03:17 PM, Gilbert Sebenste wrote:
> >>>> Here it is:
> >>>>
> >>>>
> >>>>
> >>>> NWS Statement on Cause of Outage on Feb. 13
> >>>> Feb. 14, 2017
> >>>>
> >>>> The National Weather Service experienced a failure of its AWIPS
> Network
> >>>> Control Facility communications network at 2:08 p.m. EST Feb 13. The
> >>>> outage, lasting two hours, 36 minutes, prevented us from fully
> >>>> distributing forecasts and warnings. During the outage, the public was
> >>>> able to access forecasts, watches and warnings by NOAA Weather Radio
> and
> >>>> the social media accounts of their local forecast office.
> >>>>
> >>>> Technicians quickly determined the cause of the problem was the
> >>>> simultaneous failure of two core communications routers - primary and
> >>>> backup - for the control facility due to a power problem. The routers
> >>>> were replaced and the system was restored to full service. We are
> still
> >>>> investigating what caused the power outage.
> >>>>
> >>>> The AWIPS communications system is a very reliable configuration and
> >>>> this is the first time both routers failed simultaneously.
> >>>>
> >>>> We are implementing additional communication pathways to the backup
> >>>> Network Control Facility to ensure that problems encountered in
> >>>> switching operations to this backup facility will not recur.
> >>>>
> >>>> ---
> >>>>
> >>>>
> >>>>
> >>>> *Gilbert Sebenste*
> >>>>
> >>>> Staff Meteorologist
> >>>>
> >>>>
> >>>>
> >>>> Environmental Health and Safety
> >>>>
> >>>> Labs for Wellness 154 | DeKalb, Illinois 60115
> >>>>
> >>>> 815-753-5492
> >>>>
> >>>> _gilbert@xxxxxxx <mailto:gilbert@xxxxxxx>_
> >>>>
> >>>> http://weather.admin.niu.edu <http://weather.admin.niu.edu/>
> >>>>
> >>>> Everyone. Home. Safely.
> >>>>
> >>>>
> >>>>
> >>>> NIU
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> NOTE: All exchanges posted to Unidata maintained email lists are
> >>>> recorded in the Unidata inquiry tracking system and made publicly
> >>>> available through the web. Users who post to any of the lists we
> >>>> maintain are reminded to remove any personal information that they
> >>>> do not want to be made public.
> >>>>
> >>>>
> >>>> ldm-users mailing list
> >>>> ldm-users@xxxxxxxxxxxxxxxx
> >>>> For list information or to unsubscribe, visit:
> >>>> http://www.unidata.ucar.edu/mailing_lists/
> >>>>
> >>>
> >>> _______________________________________________
> >>> NOTE: All exchanges posted to Unidata maintained email lists are
> >>> recorded in the Unidata inquiry tracking system and made publicly
> >>> available through the web. Users who post to any of the lists we
> >>> maintain are reminded to remove any personal information that they
> >>> do not want to be made public.
> >>>
> >>>
> >>> ldm-users mailing list
> >>> ldm-users@xxxxxxxxxxxxxxxx
> >>> For list information or to unsubscribe, visit:
> >>> http://www.unidata.ucar.edu/mailing_lists/
> >>>
> >>
> >>
> >>_______________________________________________
> >>NOTE: All exchanges posted to Unidata maintained email lists are
> >>recorded in the Unidata inquiry tracking system and made publicly
> >>available through the web. Users who post to any of the lists we
> >>maintain are reminded to remove any personal information that they
> >>do not want to be made public.
> >>
> >>
> >>ldm-users mailing list
> >>ldm-users@xxxxxxxxxxxxxxxx
> >>For list information or to unsubscribe, visit:
> >> http://www.unidata.ucar.edu/mailing_lists/
> >>
> >
>
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
--
Gerry Creager
NSSL/CIMMS
405.325.6371
++++++++++++++++++++++
“Big whorls have little whorls,
That feed on their velocity;
And little whorls have lesser whorls,
And so on to viscosity.”
Lewis Fry Richardson (1881-1953)
ldm-users archives: