Hi Jeff:
Thanks for clarifying this, along with why best practice is to keep previous
versions around.
Interesting example in para 2; we have seen messages that use elements from
subsequent versions. I take this to mean that producers aren't always that
careful with what version they say they're using. If the message is otherwise
well-formed, I hate to reject it. However, immediate feedback to the producer
could fix the problem.
In terms of collecting versioned tables: Unidata and the British Met Office
(Gil Ross) have independently extracted the info from the current (version 13)
WMO master tables B and D. We are comparing and reconciling any differences, so
Im feeling pretty good that we will end up with an accurate representation of
whats in those documents. We should finish that work when Gil is back from
vacation.
However, there is the problem of earlier versions. Unidata has extracted and compared the versioned mel-bufr tables. But cross-referencing with the ECMWF tables (from BUFRDC package) fails to confirm the differences. So at the moment I am having doubts about the mel-bufr tables. In short, we are still looking for a reliable way to generate earlier versions of the tables.
But the good news is that, in principle, we should be able to decode all
previous versions with the current master tables.
I hadn't fully realized that new table C operators are similar to a format
change, so the edition number has to increase. Ill have to digest that a bit
more.
BTW, if anyone knows who at the ECMWF might be interested in joining our
discussion, let me know (or just invite them).
Regards,
John
Jeff Ator wrote:
Hello Everyone,
I'm now back from a restful vacation and can add my proverbial $0.02 to
the discussion.
First of all, Stan is correct that scale, reference and bit width values
should never change between table versions. Once a descriptor has been
included in an operational version it is considered static. This means
that you theoretically only ever need the latest version of Table B in
order to be able to decode everything, and any discrepancies for a
particular descriptor between successive table versions are indeed typos.
Having said that, I do believe it is useful to maintain previous table
versions, because they do allow you to go back and determine when
certain descriptors became available for use. For example, if you
received a message that said it was using version 8 of the tables, but
it contained a particular descriptor that only became available
beginning with version 10, having all of the previous versions available
would allow you to easily diagnose such an error.
Please note that I'm only sharing with you what is the current WMO
practice, and I'm not making any personal statement of opinion one way
or the other as to whether this is a good thing. While the current
practice, whereby each successive Table B is always a superset of all
the previous versions, does simplify the task of decoding, it also has
caused some of the descriptor classes to fill up much more quickly than
would otherwise be necessary. A good example is Class 12, where we
initially had a descriptor for every conceivable type of temperature
value, but with scale 1 which only allowed accuracy to one digit beyond
the decimal. After these were in use for a while, it was discovered
that we had a problem, because most instruments report temperatures in
Celsius whereas BUFR uses the SI unit of Kelvin, but the conversion
factor between Celsius and Kelvin is 273.15, and different architectures
didn't always handle the rounding the same way. This meant you could
have one center take an observation in Celsius, then convert it to
Kelvin for encoding into BUFR, then send that BUFR message to a
different center, and the second center could, when converting back to
Celsius, obtain a value that was off by as much as two-tenths of a
degree from the original observed value. With hindsight, we realized
that the best solution was to just store all temperatures using scale 2,
but the practice was (and still is!) to never modify the characteristics
of an existing descriptor once it has been in the table. So we ended up
creating a second copy of each existing temperature descriptor, each
with it's own new reference number and a scale factor of 2, when an
alternative solution could have been to simply increase the scale of
each existing temperature descriptor effective with the next version of
the tables, and then rely on everyone's encoder/decoder software to
differentiate properly between the old and new versions of each
descriptor based on the table version number encoded within the message.
I realize that was a somewhat long-winded historical example, but an
important point is that WMO could possibly change their practice at some
point in the future (especially as some of the Table B classes are now
close to completely full!), and this is another reason why it would be a
good idea for us to hold on to all previous versions of these tables,
even though right now it's not technically necessary. The easiest way
to do this would be to include the version number somewhere in the
filename within our collective archive.
As for BUFR edition numbers (about which I've also sensed some confusion
in the ongoing email thread), these are a different animal entirely.
Whereas table version number changes do not require any corresponding
change to BUFR encoder/decoder software, the edition number is
incremented only when changes are introduced which require corresponding
changes to actual encoding and decoding software (for example, if new
Table C operators are introduced, or the format of BUFR Section 1 is
modified). In this case, the software needs to be able to
simultaneously handle multiple editions (i.e. formats) of BUFR messages,
which of course adds to programming complexity, and so such changes are
made much less frequently and with much more advance notice.
I hope this helps clarify any confusion!
Best regards,
-Jeff
