Relaying Data Via NNTP

Unidata is investigating the use of NNTP for the purpose of data relay.   An early report of the reasons for this effort and its viability is available at "On the Use of Internet News for Delivering Data" .    Some preliminary testing results are available at "Exploring Usenet as an Alternative for Distributing Geoscientific Data Via the Internet"

As an experiment, Unidata is currently relaying high resolution model output from Boulder to Washington, D.C. via INN. This is a high volume stream ranging from 23 to 25 gigabytes of data per day. A test starting on January 11 and extending for over 24 hours resulted in the successful relay of over one half million products with 99% arriving within 4 seconds or less:

       Product Latencies

Total values read: 549622
   secs  #prods   %    cumulative%
   ----  ------  ----  ------------
    -1:   1572   0.29   0.29
     0: 364983  66.41  66.69
     1: 149892  27.27  93.96
     2:  21588   3.93  97.89
     3:   5440   0.99  98.88
     4:    913   0.17  99.05
     5:    413   0.08  99.12
    10:    652   0.12  99.24
    20:   1699   0.31  99.55
    30:   1416   0.26  99.81
    40:    975   0.18  99.99
    50:     79   0.01  100.00

Graphs showing current 5 minute statistics are available "NNTP Relay Stats". These four graphs show (1) average latency per second, (2) products received per second, (3) minimum latency per second, and (4) maximum latency per second.

Products relayed via Usenet for this test are under the group unidata.* with a 'Distribution: unidata' header.   Currently all unidata.binaries.* articles will be sourced with a path header entry of imogene.unidata.ucar.edu.
 

Newsgroup Hierarchy

A newsgroup hierarchy for data relay is under development for this purpose.    That hierarchy will be posted here as groups are created.  Currently, the group being used for data relay is called unidata.binaries.grid.grib

Group Security

These are moderated groups to help ensure that the data products received are what they are represented to be.  Data products are currently checksummed to ensure proper transmission.

Data Encoding

A variety of data encodings are being evaluated for efficiency.  One, labeled as X-UnidataEncoding under the Content-Transfer-Encoding header, is a non proprietary encoding that produces output significantly smaller than some of the more standard encodings.

Viewing the Data

Unidata provides several packages for viewing atmospheric data, such as the Integrated Data Viewer (IDV), GEMPAK, and MCIDAS.  Please see our  software page  for more information about acquiring and using these packages.   However, note that some packages require  a priori "decoding" into a format suitable for viewing.  Thus, to view the data you may also need the appropriate decoders, which we also provide.  For further information about decoders please see the same page.

Note that to receive a data product via NNTP and view it, the product would need to be decoded from its transmission format and then subsequently decoded into a format compatible with the chosen visualization package.