NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

To learn about what's going on, see About the Archive Site.

Re: [ldm-users] NEXRAD2/NEXRAD3 lag?

On Sat, 10 May 2014, Blair Trosper wrote:


Specifically, we seem to be waiting on VERY the last chunk of each scan.
You should run a command like this and see when the LDM queue insert time is for each chunch of the scan: 

$ pqcat -v -l - -f NEXRD2 -p 'AMA/20140511172234' > /dev/null
May 11 17:52:40 pqcat NOTE: Starting Up (2051)
May 11 17:52:40 pqcat INFO: 8768 20140511172237.031 NEXRAD2 221001 L2-BZIP2/KAMA/20140511172234/221/1/S/V06/0 May 11 17:52:40 pqcat INFO: 114024 20140511172248.126 NEXRAD2 221002 L2-BZIP2/KAMA/20140511172234/221/2/I/V06/0 May 11 17:52:40 pqcat INFO: 120607 20140511172300.220 NEXRAD2 221003 L2-BZIP2/KAMA/20140511172234/221/3/I/V06/0 May 11 17:52:40 pqcat INFO: 125158 20140511172312.319 NEXRAD2 221004 L2-BZIP2/KAMA/20140511172234/221/4/I/V06/0 May 11 17:52:40 pqcat INFO: 112504 20140511172322.406 NEXRAD2 221005 L2-BZIP2/KAMA/20140511172234/221/5/I/V06/0 May 11 17:52:40 pqcat INFO: 137939 20140511172335.522 NEXRAD2 221006 L2-BZIP2/KAMA/20140511172234/221/6/I/V06/0 May 11 17:52:40 pqcat INFO: 118930 20140511172347.626 NEXRAD2 221007 L2-BZIP2/KAMA/20140511172234/221/7/I/V06/0 May 11 17:52:40 pqcat INFO: 48809 20140511172359.683 NEXRAD2 221008 L2-BZIP2/KAMA/20140511172234/221/8/I/V06/0 May 11 17:52:40 pqcat INFO: 49433 20140511172411.737 NEXRAD2 221009 L2-BZIP2/KAMA/20140511172234/221/9/I/V06/0 May 11 17:52:40 pqcat INFO: 54736 20140511172424.792 NEXRAD2 221010 L2-BZIP2/KAMA/20140511172234/221/10/I/V06/0 May 11 17:52:40 pqcat INFO: 49576 20140511172434.842 NEXRAD2 221011 L2-BZIP2/KAMA/20140511172234/221/11/I/V06/0 May 11 17:52:40 pqcat INFO: 57399 20140511172446.900 NEXRAD2 221012 L2-BZIP2/KAMA/20140511172234/221/12/I/V06/0 May 11 17:52:40 pqcat INFO: 49768 20140511172458.954 NEXRAD2 221013 L2-BZIP2/KAMA/20140511172234/221/13/I/V06/0 May 11 17:52:40 pqcat INFO: 65977 20140511172512.027 NEXRAD2 221014 L2-BZIP2/KAMA/20140511172234/221/14/I/V06/0 May 11 17:52:40 pqcat INFO: 76160 20140511172524.105 NEXRAD2 221015 L2-BZIP2/KAMA/20140511172234/221/15/I/V06/0 May 11 17:52:40 pqcat INFO: 82324 20140511172536.182 NEXRAD2 221016 L2-BZIP2/KAMA/20140511172234/221/16/I/V06/0 May 11 17:52:40 pqcat INFO: 79884 20140511172547.262 NEXRAD2 221017 L2-BZIP2/KAMA/20140511172234/221/17/I/V06/0 May 11 17:52:40 pqcat INFO: 81238 20140511172559.339 NEXRAD2 221018 L2-BZIP2/KAMA/20140511172234/221/18/I/V06/0 May 11 17:52:40 pqcat INFO: 68100 20140511172611.411 NEXRAD2 221019 L2-BZIP2/KAMA/20140511172234/221/19/I/V06/0 May 11 17:52:40 pqcat INFO: 26953 20140511172623.462 NEXRAD2 221020 L2-BZIP2/KAMA/20140511172234/221/20/I/V06/0 May 11 17:52:40 pqcat INFO: 31827 20140511172636.522 NEXRAD2 221021 L2-BZIP2/KAMA/20140511172234/221/21/I/V06/0 May 11 17:52:40 pqcat INFO: 34510 20140511172646.580 NEXRAD2 221022 L2-BZIP2/KAMA/20140511172234/221/22/I/V06/0 May 11 17:52:40 pqcat INFO: 33948 20140511172658.634 NEXRAD2 221023 L2-BZIP2/KAMA/20140511172234/221/23/I/V06/0 May 11 17:52:40 pqcat INFO: 33823 20140511172711.696 NEXRAD2 221024 L2-BZIP2/KAMA/20140511172234/221/24/I/V06/0 May 11 17:52:40 pqcat INFO: 27970 20140511172723.748 NEXRAD2 221025 L2-BZIP2/KAMA/20140511172234/221/25/I/V06/0 May 11 17:52:40 pqcat INFO: 49144 20140511172735.805 NEXRAD2 221026 L2-BZIP2/KAMA/20140511172234/221/26/I/V06/0 May 11 17:52:40 pqcat INFO: 56772 20140511172748.866 NEXRAD2 221027 L2-BZIP2/KAMA/20140511172234/221/27/I/V06/0 May 11 17:52:40 pqcat INFO: 60996 20140511172758.926 NEXRAD2 221028 L2-BZIP2/KAMA/20140511172234/221/28/I/V06/0 May 11 17:52:40 pqcat INFO: 60507 20140511172810.994 NEXRAD2 221029 L2-BZIP2/KAMA/20140511172234/221/29/I/V06/0 May 11 17:52:40 pqcat INFO: 56882 20140511172824.058 NEXRAD2 221030 L2-BZIP2/KAMA/20140511172234/221/30/I/V06/0 May 11 17:52:40 pqcat INFO: 50683 20140511172834.122 NEXRAD2 221031 L2-BZIP2/KAMA/20140511172234/221/31/I/V06/0 May 11 17:52:40 pqcat INFO: 20903 20140511172847.166 NEXRAD2 221032 L2-BZIP2/KAMA/20140511172234/221/32/I/V06/0 May 11 17:52:40 pqcat INFO: 24209 20140511172859.212 NEXRAD2 221033 L2-BZIP2/KAMA/20140511172234/221/33/I/V06/0 May 11 17:52:40 pqcat INFO: 25977 20140511172912.257 NEXRAD2 221034 L2-BZIP2/KAMA/20140511172234/221/34/I/V06/0 May 11 17:52:40 pqcat INFO: 26113 20140511172922.306 NEXRAD2 221035 L2-BZIP2/KAMA/20140511172234/221/35/I/V06/0 May 11 17:52:40 pqcat INFO: 24476 20140511172934.358 NEXRAD2 221036 L2-BZIP2/KAMA/20140511172234/221/36/I/V06/0 May 11 17:52:40 pqcat INFO: 21243 20140511172947.407 NEXRAD2 221037 L2-BZIP2/KAMA/20140511172234/221/37/I/V06/0 May 11 17:52:40 pqcat INFO: 55142 20140511173012.497 NEXRAD2 221038 L2-BZIP2/KAMA/20140511172234/221/38/I/V06/0 May 11 17:52:40 pqcat INFO: 58391 20140511173034.590 NEXRAD2 221039 L2-BZIP2/KAMA/20140511172234/221/39/I/V06/0 May 11 17:52:40 pqcat INFO: 55245 20140511173059.686 NEXRAD2 221040 L2-BZIP2/KAMA/20140511172234/221/40/I/V06/0 May 11 17:52:40 pqcat INFO: 48265 20140511173123.764 NEXRAD2 221041 L2-BZIP2/KAMA/20140511172234/221/41/I/V06/0 May 11 17:52:40 pqcat INFO: 48377 20140511173146.835 NEXRAD2 221042 L2-BZIP2/KAMA/20140511172234/221/42/I/V06/0 May 11 17:52:40 pqcat INFO: 48771 20140511173210.908 NEXRAD2 221043 L2-BZIP2/KAMA/20140511172234/221/43/E/V06/0 
May 11 17:52:40 pqcat NOTE: Exiting
May 11 17:52:40 pqcat NOTE: Number of products 43
So that shows the last chunk of the 17:22:34 scan being inserted at 17:32:10 

The next scan then comes ten seconds later 17:32:20
May 11 17:51:25 pqcat INFO: 8677 20140511173220.652 NEXRAD2 222001 L2-BZIP2/KAMA/20140511173217/222/1/S/V06/0
Are you able to show the above with a 10 minute lag for the last chunk? Perhaps you are seeing some irregularity with how data is getting handed off to your ingester. 

daryl


--
/**
 * Daryl Herzmann
 * Assistant Scientist -- Iowa Environmental Mesonet
 * http://mesonet.agron.iastate.edu
 */
  • 2014 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the ldm-users archives: