RE: ldm decoder

• To: Tom Baltzer <tbaltzer@xxxxxxxx>
• Subject: RE: ldm decoder
• From: Robb Kambic <rkambic@xxxxxxxxxxxxxxxx>
• Date: Wed, 7 Apr 2004 15:13:19 -0600 (MDT)

On Wed, 7 Apr 2004, Tom Baltzer wrote:

Hi again Charlie,

I'm glad to hear that we're seeing the same things now and you've got metar
data coming in as netCDF files.

You're right that metar2nc is a nontrivial script about which I know little
at this juncture.  METAR form is pretty darned cryptic, but I know that some
information about METAR can be obtained here:

http://my.unidata.ucar.edu/content/staff/awilson/public/LEAD/dataDesc/index.
html

It has references to documents that describe the form in more detail.

I've cc'd Rob Kambic, our local decoders expert who developed the metar2nc
script. Any insight/suggestions as to how to best/most quickly understand
the metar2nc script Robb?

Hiya,

Official METAR web site: http://metar.noaa.gov

Doesn't include the manual on how to decode a metar, one has to buy the
manual. I have a copy of the manual and some other hard copy printed
documentation if one would like it.

Another method would be to look up METAR decoder using google.


Robb...

Thanks,
Tom.

> -----Original Message-----
> From: Charles Moad [mailto:cmoad@xxxxxxxxxxx]
> Sent: Tuesday, April 06, 2004 10:43 PM
> To: Tom Baltzer
> Cc: 'Beth A Plale'; 'Tom Baltzer'; 'Doug Lindholm';
> machrist@xxxxxxxxxxxxxx
> Subject: RE: ldm decoder
>
>    I get identical output as well for all the mentioned feedtypes.
> They
> all say IDS|DDPLUS.
>    You were right about the missing tab on the first line, so the
> syntax
> passes now.  I am also seeing the netcdf files being created so all
> seems to be working well in that aspect.
>    The challenge we face now is turning these netcdf files into a
> stream.
> The metar2nc script is not exactly a trivial, so any input as to how it
> works might help.  We are currently using a c++ api to register a dquob
> stream source, but I think we are going to try to wrap this api in perl
> and modify the metar2nc script.
>    Thanks for you help so far.
>
> - Charlie
>
> On Tue, 2004-04-06 at 17:02, Tom Baltzer wrote:
> > Hi Charlie,
> >
> > Well that's very interesting.  My understanding is that DDPLUS is just a
> > combination of DDS and PPS (Public Products Service) so if you're able
> to
> > get DDPLUS I'd have thought you could get DDS...
> >
> > I just spoke with a Jeff Weber (a local expert) who tells me that
> IDS|DDPLUS
> > is a single feed composed of DDS, PPS and IDS and you cannot separate
> these
> > by feed type. And in fact when I run the two ldmadmin watch -f commands
> you
> > show (in two separate windows) I get identical results. I even went so
> far
> > as to try each (PPS, DDS, PPS and DDPLUS) individually in 4 windows and
> got
> > identical results.  What result do you get when you run the ldmadmin
> watch
> > with the IDS|DDS feedtype?
> >
> > As far as the error message you're getting from checking the pqact.conf
> > file, I suspect that there is yet another TAB required.  Check that the
> > white space between IDS|DDPLUS and ^S in your pqact.conf file is in fact
> a
> > tab rather than a space (sorry I didn't clarify this earlier).
> >
> > Let me know how it goes and/or if I can provide any other help.
> >
> > Best regards,
> > Tom.
> >
> >
> > > -----Original Message-----
> > > From: Charles W Moad [mailto:cmoad@xxxxxxxxxxx]
> > > Sent: Tuesday, April 06, 2004 1:23 PM
> > > To: Tom Baltzer
> > > Cc: 'Beth A Plale'; 'Charles W Moad'; 'Tom Baltzer'; 'Doug Lindholm'
> > > Subject: RE: ldm decoder
> > >
> > > As a first step I am trying run the standard pqact | metar2nc and save
> > > the data files.  When I run "ldmadmin pqactcheck" I get the following
> > > error:
> > >
> > > ---
> > > Apr 06 19:20:17 pqact[26196]: feedtype error at line 121: unknown feed
> > > name in feedtype expression: "IDS|DDPLUS ^S(A....|P....|XUS8.) ....
> > > ([0-3][0-9])"
> > > ---
> > >
> > > I don't think we get the DDS|IDS feed because it does not show up as a
> > > valid feedtype for "ldmadmin watch -f "DDS\|IDS"", however "ldmadmin
> > > watch -f "IDS\|DDPLUS"" does work.
> > >
> > > Any help would be greatly appreciated.
> > >
> > > Thanks,
> > >  Charlie
> > >
> > > On Tue, 2004-04-06 at 12:43, Tom Baltzer wrote:
> > > > Hi Beth and Charlie,
> > > >
> > > > It sounds like you have found the Unidata netCDF decoder package.
> In my
> > > > short time around here I've discovered that there are more decoders
> than
> > > > just those found in that package and have been working on
> documenting
> > > those
> > > > that I see used by LDM systems around Unidata here:
> > > > http://my.unidata.ucar.edu/content/projects/LEAD/Decoders.htm
> > > >
> > > > The pqact.conf configuration lines for pulling out metar data and
> > > placing it
> > > > in netcdf files in the ~ldm/data/pub/decoded/netcdf/surface
> directory
> > > are
> > > > (note the first 5 lines are comments):
> > > >
> > > > #
> > > > # metar decoder, creates NetCDF files formatted
> > > > #    data/decoded/netcdf/surface/yymmddhh_metar.nc
> > > > #
> > > > # All aviation reports including metar tests
> > > > DDS|IDS ^S(A....|P....|XUS8.) .... ([0-3][0-9])
> > > >         PIPE    decoders/metar2nc
> > > >         etc/metar.cdl
> > > >         data/pub/decoded/netcdf/surface
> > > >         (\2:yy)(\2:mm)
> > > >
> > > > Be advised that those are <tab>s at the beginning of lines 2-5 (very
> > > > important!).  Also be advised that it appears the decoder expects
> the
> > > > directory structure to exist and be writable before it will work.
> I've
> > > > submitted this to the decoder folks as a fix.
> > > >
> > > > As Doug mentioned the pqact application will stream the raw data
> reports
> > > to
> > > > stdin and the metar2nc perl script then takes that streaming data
> and
> > > writes
> > > > it to a netCDF file.  So if you wished to you could take the
> metar2nc
> > > script
> > > > and modify it to act as a dQUOB interface. Or if dQUOB takes stdin
> then
> > > you
> > > > could use it to help you parse the incoming metar data. My
> understanding
> > > is
> > > > that you all have been trying to access the LDM product queue to do
> your
> > > > dQUOB work.  It seems that using pqact/decoder type functions would
> be
> > > much
> > > > easier and effectively the same.
> > > >
> > > > Well I hope this helps.  Feel free to contact me directly with
> questions
> > > or
> > > > comments.
> > > >
> > > > Cheers,
> > > > Tom.
> > > >
> > > > Tom Baltzer
> > > > Software Engineer
> > > > UCAR Office of Programs
> > > > Unidata Program Center
> > > > P.O. Box 3000
> > > > Boulder, CO 80307-3000
> > > > tbaltzer@xxxxxxxx
> > > > 303-497-8637
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Doug Lindholm [mailto:lind@xxxxxxxx]
> > > > > Sent: Friday, April 02, 2004 11:52 AM
> > > > > To: Beth A Plale
> > > > > Cc: Charles W Moad; Tom Baltzer
> > > > > Subject: Re: ldm decoder
> > > > >
> > > > > Hi Beth,
> > > > >
> > > > > Tom is spearheading the LEAD related LDM and decoders setup here.
> I'm
> > > > > CC-ing him.
> > > > >
> > > > > Do you want to work with data streamed from the LDM or decoded
> files?
> > > I
> > > > > suspect you want a decoded stream, though I don't know of any
> decoders
> > > > > that do that (but then by breadth of decoder knowledge is
> limited).
> > > > >
> > > > > Since I don't understand the nature of dQUOB I'm not sure which
> data
> > > > > product would provide the path of least resistance. METAR data is
> > > simple
> > > > > conceptually but the decoding of the raw reports isn't trivial.
> You
> > > > > could modify the metar2nc decoder (perl script) to spit out
> whatever
> > > > > dQUOB wants.
> > > > >
> > > > > BTW, the METAR data comes over the IDS|DDPLUS feed. I'll let Tom
> help
> > > > > you figure out how to modify the pqact.conf file to redirect the
> METAR
> > > > > data.
> > > > >
> > > > > Let me know if you have any questions. There are a lot of
> resources at
> > > > > Unidata we could point you to.
> > > > >
> > > > > Cheers,
> > > > > Doug
> > > > >
> > > > > Beth A Plale wrote:
> > > > > > Hi Doug,
> > > > > >
> > > > > > My student, Charlie, is looking at piping data to a process, as
> you
> > > > > > described it last week, where it will be decoded then used in
> dQUOB.
> > > > > > We're looking for a good data product to use, and by "good" I
> more
> > > mean
> > > > > > that we're getting it via our LDM peer, it is relatively simple,
> and
> > > > > that
> > > > > > there's a decoder for it.  What would you suggest?  The data
> > > products
> > > > > > we're receiving are below.
> > > > > >
> > > > > > thanks!
> > > > > >
> > > > > > beth
> > > > > >
> > > > > >
> > > > > >>I don't see METAR data in the queue.  I am running "ldmadmin
> watch",
> > > and
> > > > > >>I only see CRAFT, NNEXRAD, IDS|DDPLUS, CONDUIT, HDS.
> > > > > >>
> > > > > >>- Charlie
> > > > > >
> > > > > >
> > > > > >>>   I think I found the decoder on the unidata web site.  They
> > > have
> > > > > netcdf
> > > > > >>>decoders for GRIB, METAR, Upperair, Synoptic, Buoy, Ascii, and
> > > > > >>>ldmConnect.  What data do you want me to pull from the ldm
> queue?
> > > All
> > > > > >>>of it?
> > > > > >>>
> > > > > >>>- Charlie
> > > > > >>>
> > > > > >>>
> > > > >
> > > > > --
> > > > > *-----------------------------------------------------------------
> ----
> > > -*
> > > > > | Doug Lindholm, Software Engineer          |  E-mail:
> lind@xxxxxxxx
> > > |
> > > > > | Unidata Program Center                    |   Phone: 303-497-
> 8659
> > > |
> > > > > | UCAR Office of Programs                   |
> > > |
> > > > > | P.O. Box 3000                             |     There's no place
> > > |
> > > > > | Boulder, Colorado 80307-3000              |        like $HOME
> > > |
> > > > > *-----------------------------------------------------------------
> ----
> > > -*
> > > >
> >

==============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
rkambic@xxxxxxxxxxxxxxxx                   WWW: http://www.unidata.ucar.edu/
==============================================================================