NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
I'll have to take an alternate stance to both RAID5/6 and ext4. Our experience with ext4 was not as good as ext2, but more than satisfactory. However, xfs was the order of the day when I was still at A&M. Similarly, the difference between RAID0 and RAID5 is noticable, but I had better than "just satisfactory" performance using RAID6, and fewer worries about hardware failures. I'm suspecting a time issue, offhand. gerry On Fri, May 23, 2014 at 1:13 PM, Robert Mullenax < Robert.Mullenax@xxxxxxxxxxxxx> wrote: > At this juncture it sure sounds like it almost has to be a disk > performance issue. I went from HW RAID 0 to HW RAID 5 a few years ago as a > test and was shocked to see how much slower writes are..not that I am > advocating RAID 0. We use ZFS mirroring now. > > > > -----Original Message----- > From: ldm-users-bounces@xxxxxxxxxxxxxxxx on behalf of Tyler Allison > Sent: Fri 5/23/2014 12:53 PM > To: Ryan Hickman > Cc: LDM Users > Subject: Re: [ldm-users] NEXRAD2/NEXRAD3 lag? > > I think you said you're running ext4 files system on the data directory? > > My testing of ext4 with high IO LDM servers was atrocious. We no longer > use it for our high speed disk write areas. > > If you can, mount an ext2 partition and move your writes to there for a > test . > > -Tyler > > > On May 23, 2014, at 12:34 PM, Ryan Hickman <ryan@xxxxxxxxxxxxxxxx> > wrote: > > > > Let's see your NEXRAD2 pqact entries, please. We also didn't see the > relevant 'ldmadmin watch' that Daryl suggested either so we could determine > LDM product queue insertion time. > > > > This doesn't sound like a network topology or product dissemination > issue; other list members would likely be seeing this problem, too. > However, I'd be willing to add an "allow" for NEXRAD2 from AllisonHouse's > Texas servers for 48 hours just to help you rule that out. Feel free to > email me off list to accomplish this. > > > > Otherwise I don't have many other ideas. > > > > P.S. Daryl Herzmann has been trying to help. But it seems as if you > aren't receiving his messages. > http://www.unidata.ucar.edu/mailing_lists/archives/ldm-users/2014/msg00059.html > > > > Daryl - I noticed that you might be manually removing the email quotes > in your replies. This MIGHT be throwing the Mailing List software off. If > you trace the "Follow Ups" on Blair's original message ( > http://www.unidata.ucar.edu/mailing_lists/archives/ldm-users/2014/msg00044.html), > I never see your replies. If you look at Blair's quoted messages, you'll > see that his replies only quote the "Follow Ups" and they never include > your replies. > > > > Blair - Consider adding Daryl to your email contacts. Or check your Spam > / Junk folders for his previous messages. > > > > > >> On Thu, May 22, 2014 at 8:29 PM, Blair Trosper < > blair.trosper@xxxxxxxxxxxxxxxxxxx> wrote: > >> We're still getting it from all of our servers, and I added another one > in Denver with new providers again, and still seeing the same issue. > Unique to NEXRAD2. > >> > >> Can anyone volunteer a server to feed us NEXRAD2 data for a few days > (or as a backup permanently) to try and troubleshoot this? > >> > >> None of our servers is on the same network, so it cannot be that. And > these are all rather powerful machines with extremely high connectivity and > throughput...and we don't see latency on any of the other feeds we carry. > >> > >> We're only utilizing 1-2% of our total throughput on our Chicago > server, for example, but we're getting every site 10 minutes delayed from: > >> - OU > >> - TAMU > >> - WISC > >> > >> ...but none of us four shows any lag in the topology. > >> > >> What else could be going on? > >> > >> > >>> On Sat, May 17, 2014 at 9:35 AM, Karen Cooper - NOAA Affiliate < > karen.cooper@xxxxxxxx> wrote: > >>> The level 2 data is inserted into Ldm queues at the radar RPGs as it > is generated. If it were a problem there everyone would be seeing it. > >>> > >>> Sent from me! Please pardon any tpyos > >>> > >>> > >>>> On May 17, 2014, at 1:18 AM, Blair Trosper < > blair.trosper@xxxxxxxxxxxxxxxxxxx> wrote: > >>>> > >>>> No joy there. Our providers (TAMU, OU, WISC, et al) are all under > tier 1 providers, and there is no lag. > >>>> > >>>> I can't find the document that describes how the NEXRAD2 data is fed > into the LDM system. I think it happens in DC at the Internet2 aperture, > but that information may be out of date (or I may be remembering it wrong). > >>>> > >>>> As I recall, two weeks ago, we weren't the only ones seeing this lag . > >>>> > >>>> And we're still seeing it on three servers with three separate > upstream providers -- and each server is deliberately on different > BGP/backbone providers for geographic AND network diversity. > >>>> > >>>> Something's not quite right, and I think the key is understanding how > the NEXRAD2 data gets into LDM...and then going to THAT source to find out > why it's delayed. I know Allison House's data isn't delayed (whereas > iastate's is), so my guess is they're acquiring it independently of LDM. > >>>> > >>>> So we know iastate.edu is getting data 10 minutes late just like we > are, as are our upstream providers...but the servers themselves are always > at 0 or 1 second of lag. > >>>> > >>>> Something is definitely wrong. We don't see any lag on any of the > other feed trees (HDS, IDS, WMO, NGRID, UNIWISC, NEXRAD3, etc). This seems > to be unique to NEXRAD2. > >>>> > >>>> Can someone from UCAR/UNIDATA jump in and help us...or should we open > a formal support ticket? > >>>> > >>>> > >>>>> On Sun, May 11, 2014 at 5:49 PM, Steven Emmerson < > emmerson.steven@xxxxxxxxx> wrote: > >>>>> It could be that the last NEXRAD2 data-product is stuck in a system > buffer. Try putting a "-flush" or "-close" on the relevant PIPE action. > >>>>> > >>>>>> On May 11, 2014 11:49 AM, "Gilbert Sebenste" < > sebenste@xxxxxxxxxxxxxxxxxxxxx> wrote: > >>>>>> Hi Blair, > >>>>>> > >>>>>>> We are still seeing this. I know the TOC was made aware of this > issue two > >>>>>>> weeks ago, but despite all our servers being 0 seconds lagged on > the > >>>>>>> NEXRAD2 feed tree, our data (from multiple providers who are also > not > >>>>>>> lagged) is at least 10 minutes behind. > >>>>>> > >>>>>> Been off the grid, for the most part, or I would have answered > earlier. Things to look for: > >>>>>> > >>>>>> Can you do a traceroute to the servers in question? See if anything > is > >>>>>> jammed up inbetween them and you. > >>>>>> > >>>>>> Are all network cards set at least at 100 mb? If 10 mb, it can't > handle it. > >>>>>> > >>>>>> Is the system clock correct? Are you using ntp? This is a must. > >>>>>> > >>>>>>> Also, this page is a 500 internal server error and has been for > months, and nobody has fixed it or seems to even notice: > http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_topogif?NEXRAD2 > >>>>>>> > >>>>>>> And this one takes over a minute to load: > >>>>>>> > http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_feedtree?NEXRAD2 > >>>>>>> > >>>>>>> ...but you can verify from the second link that we're not lagging > (search > >>>>>>> for "tampa" or "chicago"). > >>>>>> > >>>>>> Give those a try aboove, and see what happens. > >>>>>> > >>>>>> Gilbert > >>>>>> > >>>>>> > ******************************************************************************* > >>>>>> Gilbert Sebenste > ******** > >>>>>> (My opinions > only!) ****** > >>>>>> Staff Meteorologist, Northern Illinois > University **** > >>>>>> E-mail: sebenste@xxxxxxxxxxxxxxxxxxxxx > *** > >>>>>> web: http://weather.admin.niu.edu > ** > >>>>>> Twitter: http://www.twitter.com/NIU_Weather > ** > >>>>>> Facebook: http://www.facebook.com/niu.weather > * > >>>>>> > ******************************************************************************* > >>>>>> > >>>>>> _______________________________________________ > >>>>>> ldm-users mailing list > >>>>>> ldm-users@xxxxxxxxxxxxxxxx > >>>>>> For list information or to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/ > >>>>> > >>>>> _______________________________________________ > >>>>> ldm-users mailing list > >>>>> ldm-users@xxxxxxxxxxxxxxxx > >>>>> For list information or to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/ > >>>> > >>>> > >>>> > >>>> -- > >>>> > >>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > > >>>> Blair Trosper > >>>> Updraft Networks / Weather Data > >>>> NOC: 469-844-5440 > >>>> Early Watch Notifications: http://twitter.com/weatherwatches > >>>> _______________________________________________ > >>>> ldm-users mailing list > >>>> ldm-users@xxxxxxxxxxxxxxxx > >>>> For list information or to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/ > >> > >> > >> > >> -- > >> > >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > > >> Blair Trosper > >> Updraft Networks / Weather Data > >> NOC: 469-844-5440 > >> Early Watch Notifications: http://twitter.com/weatherwatches > >> > >> _______________________________________________ > >> ldm-users mailing list > >> ldm-users@xxxxxxxxxxxxxxxx > >> For list information or to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/ > > > > _______________________________________________ > > ldm-users mailing list > > ldm-users@xxxxxxxxxxxxxxxx > > For list information or to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/ > > > _______________________________________________ > ldm-users mailing list > ldm-users@xxxxxxxxxxxxxxxx > For list information or to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/ > -- Gerry Creager NSSL/CIMMS 405.325.6371 ++++++++++++++++++++++ “Big whorls have little whorls, That feed on their velocity; And little whorls have lesser whorls, And so on to viscosity.” Lewis Fry Richardson (1881-1953)
ldm-users archives: