NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Hi Darrel, Brian and address@hidden, As part of the investigations for duplicate NEXRAD Level III products that we are seeing on our NOAAPort receive systems, I decided to create a summary of how many "duplicate" and "retransmit" products we are seeing. The summary created on one of our NOAAPort ingest machines spans the time period of about 0Z to about 21Z today, Thursday, September 24. I put the compressed tar file of the summary stats out on our RAMADDA server for every to be able to grab: motherlode.ucar.edu/repository Unidata -> IDD -> NEXRAD Level III The summary stats are in the file named: NEXRAD Level III Duplicate Product and Retransmit Stats for 20150924 NB: - I classify products as being "duplicates" if the date field in their Product IDs are the same, and their MD5 checksums are different In this case, each product is considered to be unique, and it is inserted in the local LDM queue and redistributed in our Internet Data Distribution (IDD) system. Observation: the difference in product creation time for "duplicate" products is typically very small, varying between a few milliseconds to a small number of seconds. - I classify products as being "retransmits" if the date field in their Product IDs are the same, but their MD5 checksums are the same I this case, there already exists a product with the same MD5 checksum in the local LDM queue, so the newly ingest product is thrown away as being a duplicate; these duplicates are _not_ redistributed in the IDD. Observation: the time difference in product creation time for "retransmit" products is typically on the order of many seconds (e.g., 30+). - our LDM product queue holds about an hour of products received in the NOAAPort SBN This means duplicate products sent more than an hour after the original will never be seen to be a "duplicate" or "retransmit". The file in the compressed tar file named nexrad_dupstats.log contains the highest level summary of received "duplicate" and "retransmit" products. Here is a snippit from the file: ABC:ALL:: nDups: 18 nRtran: 44 <- ABC is the NEXRAD ID; ALL means all Level III products from the NEXRAD AMX:ALL:: nDups: 41 nRtran: 126 BMX:ALL:: nDups: 20 nRtran: 42 CLX:ALL:: nDups: 43 nRtran: 83 DMX:ALL:: nDups: 38 nRtran: 136 ESX:ALL:: nDups: 2979 nRtran: 101 FTG:ALL:: nDups: 17 nRtran: 29 ... JAX:ALL:: nDups: 1903 nRtran: 65 LRX:ALL:: nDups: 10 nRtran: 32 MKX:ALL:: nDups: 15 nRtran: 41 MXX:ALL:: nDups: 24 nRtran: 68 POE:ALL:: nDups: 10 nRtran: 31 SJT:ALL:: nDups: 35 nRtran: 88 UEX:ALL:: nDups: 18 nRtran: 108 ------------------------------------ ALL:ALL:: nDups: 43969 nRtran: 8325 <- all Level III products from all NEXRADs in the 0 - 21 Z time frame Comment: - _IF_ the way that I am classifying products as being "duplicate" or "retransmit" is correct (and I think that it is, but I am willing to be convinced otherwise), there are LOTS of "duplicates" and "retransmits" being sent in the NOAAPort SBN This seems like a very bad waste of bandwidth I welcome comments by the recipients of this email!!! Cheers, Tom -- **************************************************************************** Unidata User Support UCAR Unidata Program (303) 497-8642 P.O. Box 3000 address@hidden Boulder, CO 80307 ---------------------------------------------------------------------------- Unidata HomePage http://www.unidata.ucar.edu **************************************************************************** Ticket Details =================== Ticket ID: LDE-365303 Department: Support IDD Priority: High Status: Closed