[python-users] 20200721: Re: [ldm-users] goes-restitch breaking on current G16 Meso1 data

• To: Mike Zuranski <zuranski@xxxxxxxxxxxxxxx>
• Subject: [python-users] 20200721: Re: [ldm-users] goes-restitch breaking on current G16 Meso1 data
• From: Tom Yoksas <yoksas@xxxxxxxx>
• Date: Tue, 21 Jul 2020 12:57:21 -0600

Hi Mike and Ryan,

I just took a closer look at the LDM log messages that were included
in the original message by Mike (to python-users?  I never saw a post
to ldm-users).  I think what is going on is a typo in the pattern-action
file action that is being run.  Here is the log output I am referring
to:

The lack of the decoder name in the output strongly suggests that
there is not a tab between '-metadata' and the part immediately before
it.  Please check your pattern-action file action and correct this
typo if, of course, that it exists and then do the following:

<as 'ldm'>
ldmadmin pqactcheck

If this reports no errors:

ldmadmin pqactHUP

Cheers,

Tom

On 7/21/20 12:49 PM, Ryan May wrote:

Hi Mike,

We're not seeing any issues with our feed or execution of goes-restitch. I'm actually seeing TISU as Mesoscale-2 right now, with Mesoscale-1 under TISH. Regardless, no anomalous behavior with what we're running.

Are you seeing anything in the logs from the script itself? (i.e. /home/ldm/var/logs/goes-restitch/SU.log)

For the Mesoscale sectors, goes-restitch isn't doing much, since the sectors are generally only one tile (they are today). All that's happening is removing WMO header/footer and adjusting a bit of netCDF metadata.

The first place I'd look, then, is to see if something's going wrong with /home/scripts/sat/goes_mcidas/manager.py.

Ryan

On Tue, Jul 21, 2020 at 10:34 AM Mike Zuranski <zuranski@xxxxxxxxxxxxxxx <mailto:zuranski@xxxxxxxxxxxxxxx>> wrote:

    Greetings,

    I'm using Unidata's goes-restitch.py to merge the GOES-R tiles
    coming from NOAAPort.  It's been working well for years with nary an
    issue, until this morning...

     From what I can tell, GOES-16 Mesoscale-1 was moved over the
    Atlantic (9.5N/41W, header of TISU.. ) at around 12Z, and almost
    immediately my ldmd.log began filling up with what I'm pasting
    below.  After about an hour of that, it began to impact the rest of
    our satellite processing operation.  I've cut out G16 Meso1 from
    processing at the pqact, remade the queue & restarted LDM, and
    everything has been fine since.  But if I try to re-enable that, my
    LDM log starts filling up with those errors right away again.  I
    tried to set the goes-restitch.py logging to verbose, but no errors
    were being reported in that log file, only the ldmd.log file.

    If anyone else uses goes-restitch.py with NOAAPort tiles, are you
    seeing the same thing?  I don't store the tiles locally so I haven't
    interrogated them yet to see if anything has changed, but it sure
    seems like this script doesn't like something about these SU tiles
    coming in.

    Note: I am using a slightly modified version of goes-restitch.py to
    execute another script once a full set of tiles has been stitched
    together so it can be acted upon.  If you're wondering about the "-e
    /home/scripts/sat/goes_mcidas/manager.py -p" bit below, that's all
    it's doing.  I submitted a pull-request for this a while back, so
    the code changes can be seen here:
    https://github.com/Unidata/ldm-alchemy/pull/4. ; Again, this has
    never been an issue for the several years I've been using it.

    ldmd.log output example:

20200721T120016.942166Z pqact[59355]  pbuf.c:pbuf_flush:113               ERROR Broken pipe 20200721T120016.942279Z pqact[59355]  pbuf.c:pbuf_flush:113               ERROR Couldn't write to pipe:

    fd=1021, len=4096

20200721T120016.942298Z pqact[59355]  filel.c:pipe_put:1930               ERROR Couldn't write

    307601-byte product to pipe

20200721T120016.942317Z pqact[59355]  filel.c:pipe_out:2115               ERROR Couldn't write product

    data to pipe

20200721T120016.942335Z pqact[59355]  filel.c:pipe_prodput:2178           ERROR Couldn't pipe product to

    decoder "-metadata /home/scripts/ldm-alchemy/goes-restitch.py -q -d
    /home/data/goes -t 60 -e /home/scripts/sat/goes_mcidas/manager.py -p
    -l /home/ldm/var/logs/goes-restitch/SU.log SU 211200"

20200721T120016.942353Z pqact[59355]  filel.c:pipe_prodput:2187           ERROR Decoder terminated

    prematurely

20200721T120016.942375Z pqact[59355]  filel.c:fl_removeAndFree:425        ERROR Deleting failed PIPE

    entry: pid=66861, cmd="-metadata
    /home/scripts/ldm-alchemy/goes-restitch.py -q -d /home/data/goes -t
    60 -e /home/scripts/sat/goes_mcidas/manager.py -p -l
    /home/ldm/var/logs/goes-restitch/SU.log SU 211200"

    Thanks,

    -Mike

    ======================
    Mike Zuranski
    Meteorology Support Analyst
    College of DuPage - Nexlab
    Weather.cod.edu <http://Weather.cod.edu>
    ======================
    _______________________________________________
    NOTE: All exchanges posted to Unidata maintained email lists are
    recorded in the Unidata inquiry tracking system and made publicly
    available through the web.  Users who post to any of the lists we
    maintain are reminded to remove any personal information that they
    do not want to be made public.


    python-users mailing list
    python-users@xxxxxxxxxxxxxxxx <mailto:python-users@xxxxxxxxxxxxxxxx>
    For list information, to unsubscribe, or change your membership
    options, visit: https://www.unidata.ucar.edu/mailing_lists/



--
Ryan May, Ph.D.
Software Engineer
UCAR/Unidata
Boulder, CO

_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


ldm-users mailing list
ldm-users@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
https://www.unidata.ucar.edu/mailing_lists/

--
+----------------------------------------------------------------------+
* Tom Yoksas                                      UCAR Unidata Program *
* (303) 497-8642 (last resort)                           P.O. Box 3000 *
* yoksas@xxxxxxxx                                    Boulder, CO 80307 *
* Unidata WWW Service                     http://www.unidata.ucar.edu/ *
+----------------------------------------------------------------------+

• Follow-Ups:
  • Re: [python-users] 20200721: Re: [ldm-users] goes-restitch breaking on current G16 Meso1 data
    • From: Mike Zuranski

• References:
  • [python-users] goes-restitch breaking on current G16 Meso1 data
    • From: Mike Zuranski
  • Re: [python-users] goes-restitch breaking on current G16 Meso1 data
    • From: Ryan May