Re: [ldm-users] [python-users] goes-restitch breaking on current G16 Meso1 data

• To: Mike Zuranski <zuranski@xxxxxxxxxxxxxxx>
• Subject: Re: [ldm-users] [python-users] goes-restitch breaking on current G16 Meso1 data
• From: Ryan May <rmay@xxxxxxxx>
• Date: Tue, 21 Jul 2020 12:49:22 -0600

Hi Mike,

We're not seeing any issues with our feed or execution of goes-restitch.
I'm actually seeing TISU as Mesoscale-2 right now, with Mesoscale-1 under
TISH. Regardless, no anomalous behavior with what we're running.

Are you seeing anything in the logs from the script itself? (i.e.
/home/ldm/var/logs/goes-restitch/SU.log)

For the Mesoscale sectors, goes-restitch isn't doing much, since the
sectors are generally only one tile (they are today). All that's happening
is removing WMO header/footer and adjusting a bit of netCDF metadata.

The first place I'd look, then, is to see if something's going wrong with
/home/scripts/sat/goes_mcidas/manager.py.

Ryan

On Tue, Jul 21, 2020 at 10:34 AM Mike Zuranski <zuranski@xxxxxxxxxxxxxxx>
wrote:

> Greetings,
>
> I'm using Unidata's goes-restitch.py to merge the GOES-R tiles coming from
> NOAAPort.  It's been working well for years with nary an issue, until this
> morning...
>
> From what I can tell, GOES-16 Mesoscale-1 was moved over the Atlantic
> (9.5N/41W, header of TISU.. ) at around 12Z, and almost immediately my
> ldmd.log began filling up with what I'm pasting below.  After about an hour
> of that, it began to impact the rest of our satellite processing
> operation.  I've cut out G16 Meso1 from processing at the pqact, remade the
> queue & restarted LDM, and everything has been fine since.  But if I try to
> re-enable that, my LDM log starts filling up with those errors right away
> again.  I tried to set the goes-restitch.py logging to verbose, but no
> errors were being reported in that log file, only the ldmd.log file.
>
> If anyone else uses goes-restitch.py with NOAAPort tiles, are you seeing
> the same thing?  I don't store the tiles locally so I haven't interrogated
> them yet to see if anything has changed, but it sure seems like this script
> doesn't like something about these SU tiles coming in.
>
> Note: I am using a slightly modified version of goes-restitch.py to
> execute another script once a full set of tiles has been stitched together
> so it can be acted upon.  If you're wondering about the "-e
> /home/scripts/sat/goes_mcidas/manager.py -p" bit below, that's all it's
> doing.  I submitted a pull-request for this a while back, so the code
> changes can be seen here: https://github.com/Unidata/ldm-alchemy/pull/4.
> Again, this has never been an issue for the several years I've been using
> it.
>
> ldmd.log output example:
> 20200721T120016.942166Z pqact[59355]                pbuf.c:pbuf_flush:113
>               ERROR Broken pipe
> 20200721T120016.942279Z pqact[59355]                pbuf.c:pbuf_flush:113
>               ERROR Couldn't write to pipe: fd=1021, len=4096
> 20200721T120016.942298Z pqact[59355]                filel.c:pipe_put:1930
>               ERROR Couldn't write 307601-byte product to pipe
> 20200721T120016.942317Z pqact[59355]                filel.c:pipe_out:2115
>               ERROR Couldn't write product data to pipe
> 20200721T120016.942335Z pqact[59355]
>  filel.c:pipe_prodput:2178           ERROR Couldn't pipe product to decoder
> "-metadata /home/scripts/ldm-alchemy/goes-restitch.py -q -d /home/data/goes
> -t 60 -e /home/scripts/sat/goes_mcidas/manager.py -p -l
> /home/ldm/var/logs/goes-restitch/SU.log SU 211200"
> 20200721T120016.942353Z pqact[59355]
>  filel.c:pipe_prodput:2187           ERROR Decoder terminated prematurely
> 20200721T120016.942375Z pqact[59355]
>  filel.c:fl_removeAndFree:425        ERROR Deleting failed PIPE entry:
> pid=66861, cmd="-metadata /home/scripts/ldm-alchemy/goes-restitch.py -q -d
> /home/data/goes -t 60 -e /home/scripts/sat/goes_mcidas/manager.py -p -l
> /home/ldm/var/logs/goes-restitch/SU.log SU 211200"
>
> Thanks,
>
> -Mike
>
> ======================
> Mike Zuranski
> Meteorology Support Analyst
> College of DuPage - Nexlab
> Weather.cod.edu
> ======================
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web.  Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> python-users mailing list
> python-users@xxxxxxxxxxxxxxxx
> For list information, to unsubscribe, or change your membership options,
> visit: https://www.unidata.ucar.edu/mailing_lists/
>


-- 
Ryan May, Ph.D.
Software Engineer
UCAR/Unidata
Boulder, CO