LDM and IDD Status Report
October 2008

Steve Emmerson
Mike Schmidt
Jeff Weber
Tom Yoksas

September 22, 2008

---

LDM Status Report

LDM version of previous report was 6.6.5. Current LDM is also 6.6.5. (Beta LDM version is 6.7.0.5.)

LDM release highlights since previous report:

• Added more rpcgen(1) output to the distribution to reduce dependency on that utility.
• Added "-metadata" and "-nodata" options to the PIPE action of pqact(1).
• Improved checking of pqact(1) configuration-files by the command "ldmadmin pqactcheck": LDM configuration-file entries like "EXEC pqact -f CONDUIT" (i.e., with options but with no explicit configuration-file) are now correctly handled.
• Added checking of LDM configuration-file syntax to ldmadmin(1)'s "start" command.
• Changed behavior of pqact(1) if a corrupt state-file is read. Before, the process would exit. Now, it continues with the most recent data-product (same as if the state-file didn't exist).
• Made each decoder its own process-group leader. This should improve the performance of decoders like GEMPAK's dcgrib2(1).
• Slightly modified "configure" script to work around non-standard tr(1) utilities.
• Added code to the parser of feedtype expressions to prevent an infinite loop upon encountering two consecutive UNION operators.
• Added the typedef "prod_class_t" to the header-file "ldm.h" and changed all code and documentation to use it instead of the typedef "prod_class". This was done to allow "ldm.h" to be processed by a C++ compiler. The typedef "prod_class" still exists for backward compatibility -- though it is no longer used in the code. Both C and C++ programmers should now use "prod_class_t" instead of "prod_class".
• Removed retry attempt from write-locking fcntl() call in file "pq/pq.c" because only fatal signals could occur.
• Improved some log messages.

---

IDD Status Report

• Approximately 460 machines at 250+ sites are running LDM-6 and reporting real time statistics. Unidata staff routinely assist in the installation of LDM-6 at user sites to help evolve the IDD as rapidly as possible. There are several organizations/projects that use the LDM to move substantial amounts of data that do not report statistics to Unidata: NASA, Spain, South Korea, private compaines, etc.).

• Real time LDM statistics can be found in the Real Time IDD Statistics web page (http://www.unidata.ucar.edu/projects/idd/rtstats). The topology and data flow information reflects changes in feed requests within a few minutes of changes. rtstats continue to be most useful in diagnosing reception problems at sites, especially when sites have implemented restrictions on packets volume in and out of their campus without regard for their user needs.

• IDD toplevel relay node, idd.unidata.ucar.edu

  The cluster approach to toplevel IDD relay, has been operational at the UPC since early summer 2005. The cluster, described in the June 2005 CommunitE-letter article Unidata's New IDD Cluster, routinely relays data to more than 460 downstream connections. Data input to the cluster nodes is approx. 3.6 GB/hr (0.086 TB/day); average data output by the cluster is approx. 280 Mbps (~2.8 TB/day); peak rates routinely exceeding 600 Mpbs (~6.2 TB/day).

  The following lists the breakdown by feedtype of the data being relayed by Unidata's toplevel IDD relay, idd.unidata.ucar.edu.

  Data Volume Summary for idd.unidata.ucar.edu on 20080922.1937 [UTC]
  
  Maximum hourly volume   8923.193 M bytes/hour
  Average hourly volume   4852.700 M bytes/hour
  
  Average products per hour     191405 prods/hour  (53 prods/sec)
  
  Feed                           Average             Maximum     Products
                       (M byte/hour)            (M byte/hour)   number/hour
  CONDUIT                1847.921    [ 38.080%]     4880.452    50085.200
  NEXRAD2                1674.777    [ 34.512%]     2353.337    52681.700
  NGRID                   618.664    [ 12.749%]     1023.846    15060.267
  HDS                     217.050    [  4.473%]      429.438    19361.200
  NNEXRAD                 188.339    [  3.881%]      238.032    24334.333
  NIMAGE                  163.104    [  3.361%]      241.412      174.300
  FNEXRAD                  64.751    [  1.334%]       83.844       77.200
  IDS|DDPLUS               32.816    [  0.676%]       36.517    29224.500
  UNIWISC                  22.724    [  0.468%]       28.058       20.200
  EXP                      20.703    [  0.427%]       31.083      351.667
  FSL2                      1.605    [  0.033%]        1.704       22.667
  DIFAX                     0.188    [  0.004%]        1.517        2.367
  LIGHTNING                 0.058    [  0.001%]        0.151        9.867
  

  Currently six real server nodes operating in two separate locations on the UCAR Foothills Lab campus (in the UPC offices and in FL-2) and one director comprise idd.unidata.ucar.edu. A second and third directors have been configured for added redundancy.

  The cluster approach to IDD relay has been adopted by NOAA/GSD and Penn State (using funds provided by the Unidata-administered Equipment Awards program). The cluster operated by Penn State is scheduled to assume toplevel relay responsibilities for CONDUIT data in early October.

• Internet2 (I2) bandwidth usage by the LDM protocol continues at about 19.5 TB/week (peak usage was approx. 21 TB/week the week of August 15, 2005). Weekly I2 usage statistics can be seen at Internet NetFlow: Weekly Reports (search for 'UNIDATA LDM' usage statistics in the Advanced Applications portion of Table 7). The amount of IDD traffic being reported by I2 netflow statistics fell when UCAR and other organizations began routing preferentially over National Lambda Rail (NLR) instead of I2. More recently, a movement back to use of I2 has resulted in increased LDM netflow statistics.

• NWS continues to use LDM-6 to collect and relay NEXRAD Level II data. Results of performance monitoring studies being conducted by NOAA personnel (Blanchard) are shared with the UPC, and information gained is being used to guide code modifications that will be included in future releases.

• The NWS has finished modifying the format of NEXRAD Level II products through deployment of Build 10.0 adding message type 31 in support of higher resolution 1/4 km x 0.5 degree radar operations, as well as future dual polarization mode data. The LDM product identifiers were modified to contain additional information about the data products. The new Level II data format allows for variable length records to preserve bandwith.

  The additional data in NEXRAD Level II products measurably increased the total volume of the IDD NEXRAD2 data feed.

• The LDM-6 continues to be used for data collection activities in the THORPEX Interactive Grand Global Ensemble (TIGGE). The UPC continues to provide support as needed in the use or the LDM by TIGGE participants. TIGGE stress testing activities were reported in the February, 2006 CommunitE-letter article THORPEX/TIGGE and the LDM.

• Currently, there are 13 sites known to be running the Unidata NOAAPORT reception software including Raytheon which is doing so in support of its use of Unidata GEMPAK which is informing their development of AWIPS-2. 7 community sites are acting as IDD injection nodes for NOAAPORT data.

---

Next Generation LDM Status Report

---

For previous information, please refer to the Spring 2008 User Committee status report.