NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Jason, We experienced an issue similar to what you're describing and for us the culprit was insufficient disk I/O on the EDEX server, even with an array of eight 10K RPM 12G SAS hard disks. When a class started launching CAVE and loading data not only would their clients slow down (even menus would take time to populate), but we also saw data processing latency on EDEX start climbing until the class was done loading their initial data sets. In a few cases the EDEX server could not catch up with processing and we had to stop the LDM and give EDEX a chance to clear the backlog. Monitoring with top we saw IO-waits typically between 5% and 10%, with instances as high as 20% It's worth noting we originally had possibly the most inefficient disk set up imaginable: EDEX was running on a VM and virtual storage, so the hypervisor was dealing with two layers of file systems on a RAID 5 array. That was a lot of extra work... We replaced the hard disk array with a PCIe RAID card with two NVMe SSDs, directly attached the SSD array to the VM, and the difference was MIND BLOWING. IO-wait stays below 1% and data processing latency messages have disappeared entirely, even when a class of 30 students are using CAVE. We even saw CPU usage drop significantly, probably because very little time is being wasted waiting for read/write ops now. Robert Haley Weather Systems Administrator Applied Aviation Sciences, College of Aviation 600. S. Clyde Morris Blvd. Daytona Beach, FL 32114 386.323.8033 haley787@xxxxxxxx Embry-Riddle Aeronautical University Florida | Arizona | Worldwide -----Original Message----- From: awips2-users-bounces@xxxxxxxxxxxxxxxx <awips2-users-bounces@xxxxxxxxxxxxxxxx> On Behalf Of Kaiser, Jason N. Sent: Wednesday, April 3, 2019 12:36 PM To: Michael James <mjames@xxxxxxxx> Cc: awips2-users@xxxxxxxxxxxxxxxx Subject: [EXTERNAL] Re: [awips2-users] EDEX to CAVE latency with multiple simultaneous users Hi Michael, /awips2/cave/ is locally mounted, on each SSD. Only the home directories are NFS-mounted. Multiple sessions of CAVE are run as different users (i.e. students are each logged in to Linux with their own user account), meaning that you’re correct, no two users should be reading/writing to the same ~/caveData directory at the same time. I will try turning off data cacheing and see if that alleviates the problem. -Jason From: Michael James <mjames@xxxxxxxx> Sent: Wednesday, April 3, 2019 10:52 AM To: Kaiser, Jason N. <jason.kaiser@xxxxxxxxxxxxxxxxxxx> Cc: awips2-users@xxxxxxxxxxxxxxxx Subject: Re: [awips2-users] EDEX to CAVE latency with multiple simultaneous users Hi Jason, I don't believe that CAVE using an NFS-mounted user home directory should result in the performance issues you are experiencing, but I wonder if multiple users running the same CAVE executable over NFS could cause this... is that how the application is being used? (meaning /awips2/cave/ is on an NFS mount and each users is running the app from that mount?). In our classrooms we have seen no issues with multiple CAVE clients connecting to a single server and I have not seen network latency caused by multiple clients connecting at the same time. Can we confirm that the multiple session of CAVE are run as different users, meaning no two users would be reading/writing the same ~/caveData directory at the same time? Perhaps turning off data cacheing (CAVE > Preferences > Cache) would reduce the latency to an acceptable level? _______________________________________________ NOTE: All exchanges posted to Unidata maintained email lists are recorded in the Unidata inquiry tracking system and made publicly available through the web. Users who post to any of the lists we maintain are reminded to remove any personal information that they do not want to be made public. awips2-users mailing list awips2-users@xxxxxxxxxxxxxxxx For list information, to unsubscribe, or change your membership options, visit: http://www.unidata.ucar.edu/mailing_lists/
awips2-users archives: