NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.

[bufrtables] More on table versions

Hi Stan, et al:
I previously reported changes in various versions of the mel-bufr tables which led to thinking BUFR descriptors in the WMO master tables were changing between versions (previous posts copied below, for completeness). Now Im looking at the ECMWF tables used in the BUFRDC encoding/decoding software. They fail to confirm the differences in the mel-bufr table. This leads me to suspect the accuracy of the mel-bufr tables. So perhaps (hopefully), Stan, you are right that elements cant change once they become operational. Still, I wonder why mel-bufr and ECMWF keep versioned copies of the tables?
Also what to make of element 0-22-39 in the latest Table B document (http://www.wmo.int/pages/prog/www/WMOCodes/Operational/BUFR/BufrTabB-11-2007.pdf) which has changed the bit width to 13, when it has been 12 all along? Must be a typo; is anyone using that value(!) ?
ECMWF tables have a few changes of their own between versions. It could be that they are typos, that im looking at "pre-operational" values, or things that are local to ECMWF.
The lack of table management is quite a burden on developing a general reading package. 
To reiterate what we are thinking here at Unidata:
In the absence of a better mechanism, we will publish on the Unidata web site the BUFR tables we are using for our BUFR reading software, in various standard formats like XML and ASCII, in the hopes that any mistakes in them will be quickly caught and fixed. We will try to obtain canonical sources and have tools to compare representations to check for errors. We will encourage anyone else producing BUFR messages intended for data exchange (as opposed to internal consumption only) to send us their tables or links to their tables, which we will make available on our web site.
As soon as WMO or any other authoritative organization is willing and able to take over these tasks, and publish canonical machine-readable BUFR tables, we will point everyone to that site(s) instead. 

John

PS: Please let me know if you'd like to be removed from this list, or go to 
http://www.unidata.ucar.edu/mailing_lists/



-----Original Message-----
From: John Caron [mailto:caron@xxxxxxxxxxxxxxxx] Sent: 05 August 2008 20:03 
To: Kellett, Stanley
Cc: Wright, Bruce; Jeff Ator; Ross, Gil; Michelle Mainelli; Scott
Jacobs; Paul Hamer; Chris Macdermaid; Robb Kambic
Subject: Re: table versions

Hi Stan:

Thanks so much for your reply. I hope the flies arent biting as hard as
in Wyoming where I was camping last week.

If new versions of the WMO master table can only add new elements, this
would be very good news. It would mean that one really only needs the
latest version of the table. Is there somewhere that is explicitly
stated in the WMO BUFR specification?

The worry that I have is that if a mistake is made in a published
version of a table, and is used to code a BUFR message, that message
will be incorrectly read by a decoder using the corrected version of the
table. Theres no way to unambiguously know what table the coder used. 
We have been studying the differences between versions of the tables,
using the mel-bufr tables. While we are not sure of their exact
provenance, they seem to be widely used. 
Below is a summary of the differences in scale,offset and/or bit width
among mel-bufr versions 2 through 13. The difference is always with the
current version (13); "B3M-000-003-B.diff" for example means it is the
difference with mel-bufr table version 3. For example the line 
0-2-42
 width 6 != 2 (B2M-000-002-B.diff)

means that the bit width for field 0-2-42 was 6 for version 2, and 2 for
version 13.

--------------------------------

0-2-42
 width 6 != 2 (B2M-000-002-B.diff)
 width 6 != 2 (B3M-000-003-B.diff)
 width 6 != 2 (B3M-000-004-B.diff)
 width 6 != 2 (B3M-000-005-B.diff)

0-2-91
 scale 0 != 4 (B3M-000-008-B.diff)
 refer 4 != 0 (B3M-000-008-B.diff)
 scale 0 != 4 (B3M-000-009-B.diff)
 refer 4 != 0 (B3M-000-009-B.diff)
 scale 0 != 4 (B3M-000-010-B.diff)
 refer 4 != 0 (B3M-000-010-B.diff)

0-2-140
 scale 2 != 0 (B2M-000-002-B.diff)
 scale 2 != 0 (B3M-000-003-B.diff)
 scale 2 != 0 (B3M-000-004-B.diff)
 scale 2 != 0 (B3M-000-005-B.diff)

0-2-151
 width 4 != 11 (B3M-000-006-B.diff)
 width 4 != 11 (B3M-000-007-B.diff)
 width 4 != 11 (B3M-000-008-B.diff)
 width 4 != 11 (B3M-000-009-B.diff)

0-7-9
 refer 0 != -1000 (B3M-000-008-B.diff)

0-10-9
 refer 0 != -1000 (B3M-000-008-B.diff)

0-11-52
 width 14 != 13 (B3M-000-008-B.diff)

0-11-82
 width 13 != 14 (B3M-000-008-B.diff)

0-13-60
 refer -10 != -1 (B3M-000-008-B.diff)
 refer -10 != -1 (B3M-000-009-B.diff)
 refer -10 != -1 (B3M-000-010-B.diff)

0-21-85
 refer -3000 != 0 (B2M-000-002-B.diff)
 refer -3000 != 0 (B3M-000-003-B.diff)
 refer -3000 != 0 (B3M-000-004-B.diff)
 refer -3000 != 0 (B3M-000-005-B.diff)
 refer -3000 != 0 (B3M-000-006-B.diff)

0-22-39
 width 12 != 13 (B2M-000-002-B.diff)
 width 12 != 13 (B3M-000-003-B.diff)
 width 12 != 13 (B3M-000-004-B.diff)
 width 12 != 13 (B3M-000-005-B.diff)
 width 12 != 13 (B3M-000-006-B.diff)
 width 12 != 13 (B3M-000-007-B.diff)
 width 12 != 13 (B3M-000-008-B.diff)
 width 12 != 13 (B3M-000-009-B.diff)
 width 12 != 13 (B3M-000-010-B.diff)
 width 12 != 13 (B3M-000-011-B.diff)
 width 12 != 13 (B3M-000-012-B.diff)

0-24-1
 scale -1 != -11 (B2M-000-002-B.diff)
 scale -1 != -11 (B3M-000-003-B.diff)
 scale -1 != -11 (B3M-000-004-B.diff)
 scale -1 != -11 (B3M-000-005-B.diff)

0-24-2
 scale -1 != -11 (B2M-000-002-B.diff)
 scale -1 != -11 (B3M-000-003-B.diff)
 scale -1 != -11 (B3M-000-004-B.diff)
 scale -1 != -11 (B3M-000-005-B.diff)

0-25-50
 scale 0 != 4 (B2M-000-002-B.diff)
 refer 0 != -131072 (B2M-000-002-B.diff)  width 7 != 18
(B2M-000-002-B.diff)  scale 0 != 4 (B3M-000-003-B.diff)  refer 0 !=
-131072 (B3M-000-003-B.diff)  width 7 != 18 (B3M-000-003-B.diff)  scale
0 != 4 (B3M-000-004-B.diff)  refer 0 != -131072 (B3M-000-004-B.diff)
width 7 != 18 (B3M-000-004-B.diff)  scale 0 != 4 (B3M-000-005-B.diff)
refer 0 != -131072 (B3M-000-005-B.diff)  width 7 != 18
(B3M-000-005-B.diff)  scale 0 != 4 (B3M-000-006-B.diff)  refer 0 !=
-131072 (B3M-000-006-B.diff)  width 7 != 18 (B3M-000-006-B.diff)

0-25-51
 width 12 != 7 (B2M-000-002-B.diff)
 width 12 != 7 (B3M-000-003-B.diff)
 width 12 != 7 (B3M-000-004-B.diff)
 width 12 != 7 (B3M-000-005-B.diff)
 width 12 != 7 (B3M-000-006-B.diff)

0-29-2
 width 2 != 3 (B2M-000-002-B.diff)
 width 2 != 3 (B3M-000-003-B.diff)
 width 2 != 3 (B3M-000-004-B.diff)
 width 2 != 3 (B3M-000-005-B.diff)

If we believe these mel-bufr tables (and we havent made some mistake in
analysing them), scale/reference/width seems to be allowed to change.
For example, the last field, 0-29-2, bit width was changed from 2 to 3
at version 6. If so, then we have no choice but to maintain versioned
code tables, and assume that the version number (BUFR section 1 octet
11) is correct.

Of course the real question is, what values did the coders use? We have
some hope of automatically detecting when bit width is wrong by counting
bits in the message and comparing to the data size. If scale/offset is
wrong, it will take humans and/or more complex algorithms to detect
that. 
It would be very helpful to hear what the actual practice is with tables
at the various generating centers. Or to hear that the mel-bufr tables
are wrong, or some other flaw in our conclusion.

Thanks for any thoughts,
John



Kellett, Stanley wrote:

John,
First of all I am sorry for late response as I have just gotten back from a camp trip with my family. 

No scale, width and reference values can't be changed once
operational.
What has probably happened is there was a mistake in the last published tables which was corrected. 

Stan

-----Original Message-----
From: John Caron [mailto:caron@xxxxxxxxxxxxxxxx]
Sent: 24 July 2008 18:20
To: Wright, Bruce; Jeff Ator; Ross, Gil; Michelle Mainelli; Scott Jacobs; Kellett, Stanley; Paul Hamer; Chris Macdermaid; Robb Kambic 
Subject: table versions
We are trying to understand how the different versions of the WMO master table differ from each other. We are looking at the "mel-bufr" tables, which seem to be the current best representations of the different versions of the WMO master tables, although we are not sure how they were generated.
We compare these tables against our version of the WMO master table version 13, which we extracted from the Word doc. Typical are the following difference of table 12 with table 13: 


1) *** WARNING **** Discrepency
Scale Ref Width Units for key 0-13-81
3;0;14;S m-1 from B3M-000-012-B
3;0;14;Siemens m-1 from B4M-000-013-B

2) *** WARNING **** Discrepency
Scale Ref Width Units for key 0-15-11
3;14000;13;log from B3M-000-012-B
3;14000;13;log(1/m2) from B4M-000-013-B

3) *** WARNING **** Discrepency
Scale Ref Width Units for key 0-22-39
3;-5000;12;m from B3M-000-012-B
3;-5000;13;m from B4M-000-013-B


Difference 1) and 2) seems to be clarifications of the units. In 1) "S



m-1" has been  changed to "Siemens m-1" and in 2) "log" was changed to
"log(1/m2)". It seems likely that those changes are clarifications or corrections that should be used in version 12 BUFR messages.
Difference 3) is a change in bit width of field 0-22-39. Should this be understood that version 12 messages use bit width 12, while version 


13 uses bit width 13? Obviously this is a critical thing to know.

When we compare table 10 with 13, we see:

4) *** WARNING **** Discrepency
Scale Ref Width Units for key 0-2-91
0;4;10;a from B3M-000-010-B
4;0;10;a from B4M-000-013-B

5) *** WARNING **** Discrepency
Scale Ref Width Units for key 0-13-60
1;-10;17;kg m-2 from B3M-000-010-B
1;-1;17;kg m-2 from B4M-000-013-B
For difference 4) I might guess that the scale and reference were mistakenly flipped, that is, it corrects a typo in version 10. Possibly difference 5), which changes the reference value from -10 to -1, corrects a typo also?
Are new table versions really allowed to change Scale,Ref,Width or Units of existing table entries? Because if they are, we really have to carefully track what changes with each version.
Are there archives of previous versions of WMO master tables? Are the mel-bufr tables accurate? 

Any thoughts or advice would be appreciated.