NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
I see that I'm getting a message in gnome-system-monitor regarding futex wait. I then did a search and found there is an issue with Haswell Processors (which I have)and CentOS 6.6, and that futex will hang. Brian From: Magnuson.Erik [mailto:Magnuson.Erik@xxxxxxxxx] Sent: Tuesday, March 29, 2016 12:15 PM To: Brian Bernard <bbernard@xxxxxxxxxxxx>; awips2-users@xxxxxxxxxxxxxxxx Subject: RE: High Latency in Ingest Logs I'm suggesting that you run some of the normal system performance analysis tools (top, systat, etc) see if there is something obvious like: * Excessive CPU time * Excessive "sys" time in top * I/O saturation of one of the disk devices https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide Certainly for older builds and older hardware, AWIPS2 on a single system seemed to be mostly I/O limited. (AWIPS1 as well.) While you have better hardware, the NOAAport feed and pqact.conf have also been increasing. The NWS runs AWIPS2 on distributed HW which may help balance some of the I/O load. On older HW, I ended up moving postgres and pypies to a different machine than EDEX/LDM which helped. -- Erik From: Brian Bernard [mailto:bbernard@xxxxxxxxxxxx] Sent: Tuesday, March 29, 2016 11:25 AM To: Magnuson.Erik <Magnuson.Erik@xxxxxxxxx<mailto:Magnuson.Erik@xxxxxxxxx>>; awips2-users@xxxxxxxxxxxxxxxx<mailto:awips2-users@xxxxxxxxxxxxxxxx> Subject: RE: High Latency in Ingest Logs Hi Erik, The system I'm running EDEX Server has 64GB of RAM and four 256GB SSD. CentOS 6.6 is the operating system. Would doing a system update help this issue. (That is, could it be an issue with the kernel?) Brian From: Magnuson.Erik [mailto:Magnuson.Erik@xxxxxxxxx] Sent: Tuesday, March 29, 2016 11:09 AM To: Brian Bernard <bbernard@xxxxxxxxxxxx<mailto:bbernard@xxxxxxxxxxxx>>; awips2-users@xxxxxxxxxxxxxxxx<mailto:awips2-users@xxxxxxxxxxxxxxxx> Subject: RE: High Latency in Ingest Logs Another possible reason for this would be some resource starvation of that ingest process: ? Excessive memory, swapping ? I/O bottlenecks (slow filesystem, large directory traversal w/some filesystems) ? Insufficient threads in the pool for number of items to be processed -- Erik From: awips2-users-bounces@xxxxxxxxxxxxxxxx<mailto:awips2-users-bounces@xxxxxxxxxxxxxxxx> [mailto:awips2-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Brian Bernard Sent: Tuesday, March 29, 2016 9:19 AM To: awips2-users@xxxxxxxxxxxxxxxx<mailto:awips2-users@xxxxxxxxxxxxxxxx> Subject: [awips2-users] High Latency in Ingest Logs Hi, I've been checking the ingest logs, to see what has been causing a frustrating problem where my ingestGrib stops after a couple of days, and noticed in many of the observation products such as raob and synop that I have a very high latency. Here is an example: INFO 2016-03-29 12:57:10,095 [Ingest.sfcobs-1] Ingest: EDEX: Ingest - sfcobs:: /awips2/data_store/maritime/20160329/12/SXUS23_KWNB_291200_469498449.2016032912 processed in: 2.2350 (sec) Latency: 50.6560 (sec) INFO 2016-03-29 13:08:05,361 [Ingest.goessounding-1] Ingest: EDEX: Ingest - goessounding:: /awips2/data_store/goessounding/JUTX06_KNES_291306_469516938.bufr.2016032913 processed in: 0.0030 (sec) Latency: 67.1950 (sec) INFO 2016-03-29 13:08:06,210 [Ingest.goessounding-1] Ingest: EDEX: Ingest - goessounding:: /awips2/data_store/goessounding/JUTX06_KNES_291306_469516939.bufr.2016032913 processed in: 0.0030 (sec) Latency: 68.0450 (sec) INFO 2016-03-29 13:08:10,327 [Ingest.goessounding-1] Ingest: EDEX: Ingest - goessounding:: /awips2/data_store/goessounding/JUTX06_KNES_291306_469516940.bufr.2016032913 processed in: 0.0030 (sec) Latency: 72.1610 (sec) INFO 2016-03-29 13:08:14,203 [Ingest.goessounding-1] Ingest: EDEX: Ingest - goessounding:: /awips2/data_store/goessounding/JUTX06_KNES_291306_469516941.bufr.2016032913 processed in: 0.0040 (sec) Latency: 76.0380 (sec) >From presusing an older document that Michael James put together, the issue >with high latency is due QPID. Should I re-install QPID? Thanks, Brian Bernard ________________________________ The information contained in this email message is intended only for the use of the individual(s) to whom it is addressed and may contain information that is privileged and sensitive. If you are not the intended recipient, or otherwise have received this communication in error, please notify the sender immediately by email at the above referenced address and note that any further dissemination, distribution or copying of this communication is strictly prohibited. The U.S. Export Control Laws regulate the export and re-export of technology originating in the United States. This includes the electronic transmission of information and software to foreign countries and to certain foreign nationals. Recipient agrees to abide by these laws and their regulations -- including the U.S. Department of Commerce Export Administration Regulations and the U.S. Department of State International Traffic in Arms Regulations -- and not to transfer, by electronic transmission or otherwise, any content derived from this email to either a foreign national or a foreign destination in violation of such laws.
awips2-users archives: