20050113: dcmetr... input file length limit?

• To: Mike Hardiman <Mike.Hardiman@xxxxxxxx>
• Subject: 20050113: dcmetr... input file length limit?
• From: Steve Chiswell <chiz@xxxxxxxxxxxxxxxx>
• Date: Thu, 13 Jan 2005 07:06:54 -0700 (MST)

Mike,

Since the decoders read from STDIN (eg the piped or redirected file input),
they don't see the file on disk, so are not effected by input file size.

However, the decoders such as dcmetr have a maximum bulletin length, which
is the maximum length of a product  between the ZCZC and NNNN delimeters.
For dcmetr, this limit is 100K (eg DCMXBF  = 102400 ). You will need to
create bulletin delimeters which signal to the decoder that the current
product has been read, and can be decoded, after which the decoder will search
for the next product ZCZC/NNNN pair.

Steve Chiswell
Unidata User Support

On Thu, 13 Jan 2005, Mike Hardiman wrote:

Hi Folks,

We recently acquired the Surface Archives CDs from Weather Graphics
Tecnhologies down here for use in a local research project.

Since I'd like to create a few billion surface plots, I need to get the
data into a gempak-friendly format.

So I wrote a script to read an input "laundry list" then dig into the
archive CDs, pull out the desired data, unzip it, reformat it with the
proper AFOS headers that the GEMPAK decoders so love, spit it out in
text files for each day/hour requested, and then create a GEMPAK
surface file for each day/hour using dcmetr.

However, I noticed it was only producing the GEMPAK surface files for a
few scattered days and hours.

Lo and behold, there was plenty of data in the text files that weren't
being decoded with dcmetr.

So I tried it manually... dcmetr fails.  Log just says it did not
decode any bulletins.

If I manually chomp out a huge chunk of obs from the files, they decode
just fine.... so this leads me to the following conclusion:  There must
be a input file length-limit in dcmetr.  Once it reaches this limit, it
reads no further (and therefore doesn't see the "NNNN" it needs to work
properly) and therefore fails.

Can anyone confirm my fears and/or suggest a solution?

If you need more info, like sample files, log file output, etc, please
let me know and I'll post them on the web.

Thanks much!

-Mike Hardiman
NWS Santa Teresa, NM

• References:
  • dcmetr... input file length limit?
    • From: Mike Hardiman