NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
<html><head></head><body><img src="cid:ncoEmailLogo.png@SNC.f19f4f18a72fae15" height="50"> <b>INC0316836 has new comments ( GOES Imagery)(Active)</b><hr> <style>.hideInEmail { display: none; }; </style> <div></div> <div><p>Some details of the ticket are included below:</p><table><tbody style="vertical-align:top;"><tr><th style="min-width:150px;"></th><th></th></tr><tr><td><b>End User:</b></td><td>NCO Ops</td></tr><tr><td><b>Short description:</b></td><td> GOES Imagery</td></tr><tr><td><b>Customer Impact Statement:</b></td><td>missing tiles in GOES imagery</td></tr><tr><td><b>State:</b></td><td>Active</td></tr><tr><td><b>Priority:</b></td><td>High</td></tr><tr><td><b>Category:</b></td><td>General</td></tr><tr><td><b>Sub-Category:</b></td><td>Other</td></tr><tr><td><b>Assignment group:</b></td><td>NCO Help Desk Triage</td></tr><tr><td><b>Description:</b></td><td>Received email notification from Jeff Kelley: Hi NCO. Have you been in contact with NESDIS at all about the missing tiles in GOES-16? Every image that comes in has missing tiles, whether in AWIPS, the FAA weather system, or even the College of Dupage website. <br><br>https://weather.cod.edu/satrad/?parms=continental-conus-13-24-1-100-1&checked=map&colorbar=undefined<br><br>I haven't seen any ADA or ADM messages either.<br><br>Jeff<br>ZID CWSU</td></tr></tbody></table></div> <div><p>The following comments were given about the Incident:</p><div style="margin-left:30px;"><table style="width:100%;"><tbody><tr><td><div style="padding:10px 15px; border:1px solid lightgray; border-left:7px solid Purple;margin:2px 0;"><strong>2021-12-13 20:14:24 EST - Guest </strong><sup style="float:right;">Additional comments</sup><div><span style="word-wrap:break-word;display:block;">reply from: gilbert@xxxxxxxxxxxxxxxx<br><br>Someone from the College of DuPage reached out to me with an interesting observation from the University of Oklahoma...<br><br>We had an outage on our NOAAport receive system this morning, which caused our systems to fail over to our partner organization's feeds. On investigation, we found that our receiver, ingest software, and LDM software were all working fine. We saw packets coming into the Linux-based ingest server, but nothing was being ingested. Then, it was noticed that the source IP of the NOAAport multicast traffic had changed, for at least some of the packets. Whereas in the past, it had been 10.0.9.51 , we are now seeing packets from 10.0.9.52 . Our ingest server was not accepting those because it had no special routing rule for that IP. We added such a rule (and did *not* delete the existing one for the .51 IP) and our NOAAport ingest began working as before.<br><br>I can confirm the source IP for me is 10.0.9.52 as well, but unlike him that IP range was already open on my system and that traffic was making its way in. I'm not sure if this is related or not, but I wanted to pass along the intel in case it is.</span></div></div></td></tr><tr><td><div style="padding:10px 15px; border:1px solid lightgray; border-left:7px solid Purple;margin:2px 0;"><strong>2021-12-13 19:54:36 EST - Jeff Kelley</strong><sup style="float:right;">Additional comments</sup><div><span style="word-wrap:break-word;display:block;">reply from: jeff.kelley@xxxxxxxx<br><br>I just got off the phone with NCF. They are aware that data dropouts are<br>occurring (with all data types, not just satellite) after switching between<br>systems earlier today. The tech mentioned that it might be a firewall<br>issue.<br><br>Jeff</span></div></div></td></tr><tr><td><div style="padding:10px 15px; border:1px solid lightgray; border-left:7px solid Purple;margin:2px 0;"><strong>2021-12-13 19:53:01 EST - Shellene Cope</strong><sup style="float:right;">Additional comments</sup><div><span style="word-wrap:break-word;display:block;">Gilbert Sebenste emailed: 1) I have been using the GOES Restitch for a couple of months now and it has worked perfectly until today. Today I am getting partial images. Upon looking at the log files I am finding that it is "Timing Out" waiting for the tile to arrive. I am not seeing any large amounts of dropped packets. I am not sure what could be causing my sudden issue. Any insight as to what I might check? <br> <br> <br> <br>Thanks, Jim ( partial log below) <br> <br> <br> <br> <br> <br>[TIRE 132051] 2021-12-13 20:54:52,378 [process_items]: OR_ABI-L2-CMIPC-M6C05_G16_s20213472051170_e20213472051170_c20213472051170.nc: 8 of 15 tiles received. <br> <br>[TIRE 132051] 2021-12-13 20:54:53,177 [process_items]: OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc: 7 of 15 tiles received. <br> <br>[TIRE 132051] 2021-12-13 20:54:54,033 [process_items]: OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc: 8 of 15 tiles received. <br> <br>[TIRE 132051] 2021-12-13 20:55:00,091 [process_items]: OR_ABI-L2-CMIPC-M6C03_G16_s20213472051170_e20213472051170_c20213472051170.nc: 11 of 15 tiles received. <br> <br>[TIRE 132051] 2021-12-13 20:55:00,753 [process_items]: OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc: 9 of 15 tiles received. <br> <br>[TIRE 132051] 2021-12-13 20:55:01,554 [process_items]: OR_ABI-L2-CMIPC-M6C03_G16_s20213472051170_e20213472051170_c20213472051170.nc: 12 of 15 tiles received. <br> <br>[TIRE 132051] 2021-12-13 20:55:06,066 [process_items]: OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 6 of 15 tiles received. <br> <br>[TIRE 132051] 2021-12-13 20:55:06,350 [process_items]: OR_ABI-L2-CMIPC-M6C03_G16_s20213472051170_e20213472051170_c20213472051170.nc: 13 of 15 tiles received. <br> <br>[TIRE 132051] 2021-12-13 20:55:07,089 [process_items]: OR_ABI-L2-CMIPC-M6C05_G16_s20213472051170_e20213472051170_c20213472051170.nc: 9 of 15 tiles received. <br> <br>[TIRE 132051] 2021-12-13 20:55:07,592 [process_items]: OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc: 10 of 15 tiles received. <br> <br>[TIRE 132051] 2021-12-13 20:55:08,514 [process_items]: OR_ABI-L2-CMIPC-M6C05_G16_s20213472051170_e20213472051170_c20213472051170.nc: 10 of 15 tiles received. <br> <br>[TIRE 132051] 2021-12-13 20:55:13,149 [process_items]: OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 7 of 15 tiles received. <br> <br>[TIRE 132051] 2021-12-13 20:55:16,471 [process_items]: OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 8 of 15 tiles received. <br> <br>[TIRE 132051] 2021-12-13 20:55:22,188 [process_items]: OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 9 of 15 tiles received. <br> <br>[TIRE 132051] 2021-12-13 20:55:34,664 [process_items]: OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 10 of 15 tiles received. <br> <br>[TIRE 132046] 2021-12-13 20:55:44,834 [process_items]: Finishing due to timeout. <br> <br>[TIRE 132046] 2021-12-13 20:55:44,944 [store_done]: GOES16/Products/CloudAndMoistureImagery/CONUS/Channel07/20211213/OR_ABI-L2-CMIPC-M6C07_G16_s20213472046170_e20213472046170_c20213472046170.nc finished. <br> <br>[TIRE 132046] 2021-12-13 20:55:44,947 [process_items]: Finishing due to timeout. <br> <br>[TIRE 132046] 2021-12-13 20:55:45,051 [store_done]: GOES16/Products/CloudAndMoistureImagery/CONUS/Channel12/20211213/OR_ABI-L2-CMIPC-M6C12_G16_s20213472046170_e20213472046170_c20213472046170.nc finished. <br> <br>[TIRE 132051] 2021-12-13 20:55:45,698 [process_items]: OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 11 of 15 tiles received. <br> <br>[TIRE 132046] 2021-12-13 20:55:47,772 [process_items]: Finishing due to timeout. <br> <br>[TIRE 132046] 2021-12-13 20:55:47,875 [store_done]: GOES16/Products/CloudAndMoistureImagery/CONUS/Channel10/20211213/OR_ABI-L2-CMIPC-M6C10_G16_s20213472046170_e20213472046170_c20213472046170.nc finished. <br> <br>[TIRE 132046] 2021-12-13 20:55:48,075 [process_items]: Finishing due to timeout. <br> <br>[TIRE 132046] 2021-12-13 20:55:48,181 [store_done]: GOES16/Products/CloudAndMoistureImagery/CONUS/Channel11/20211213/OR_ABI-L2-CMIPC-M6C11_G16_s20213472046170_e20213472046170_c20213472046170.nc finished. <br> <br>[TIRE 132046] 2021-12-13 20:55:49,455 [process_items]: Finishing due to timeout. <br> <br>[TIRE 132046] 2021-12-13 20:55:49,559 [store_done]: GOES16/Products/CloudAndMoistureImagery/CONUS/Channel09/20211213/OR_ABI-L2-CMIPC-M6C09_G16_s20213472046170_e20213472046170_c20213472046170.nc finished. <br> <br>[TIRE 132046] 2021-12-13 20:55:49,560 [process_items]: Finishing due to timeout. <br> <br> <br> <br>[TIRE 132046] 2021-12-13 20:55:49,664 [store_done]: GOES16/Products/CloudAndMoistureImagery/CONUS/Channel08/20211213/OR_ABI-L2-CMIPC-M6C08_G16_s20213472046170_e20213472046170_c20213472046170.nc finished. <br> <br>[TIRE 132051] 2021-12-13 20:55:49,891 [process_items]: OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 12 of 15 tiles received. <br> <br>[TIRE 132046] 2021-12-13 20:55:51,447 [process_items]: Finishing due to timeout. <br> <br>[TIRE 132046] 2021-12-13 20:55:51,556 [store_done]: GOES16/Products/CloudAndMoistureImagery/CONUS/Channel13/20211213/OR_ABI-L2-CMIPC-M6C13_G16_s20213472046170_e20213472046170_c20213472046170.nc finished. <br> <br>2) Hi Jim, <br> <br>I was just about to write my own email about this. It seems to me that not all GOES-16 tiles are making it to NOAAPort. I was about to ask if anyone else is seeing this, and I guess you are. <br> <br>I noticed this began around 14Z today. GOES-16 CONUS & Full-Disk are impacted, Mesoscale and GOES-17 seem okay. I haven't seen any announcements about data flow issues. Using notifyme I saw that Unidata was missing the same tiles I was. <br> <br>Does anyone else have any insight? <br> <br>-Mike <br> <br>3) Hello NCO, <br> <br>Some GOES-16 tiles on NOAAport/SBN are missing. This started just after 14Z, <br>when you awitched to the BMGS. See the thread below from a UNIDATA user log. <br>Multiple people are seeing this. <br> <br>Gilbert Sebenste <br>Meteorological Consultant <br>AllisonHouse, LLC <br> <br>(SEC)</span></div></div></td></tr><tr><td><div style="padding:10px 15px; border:1px solid lightgray; border-left:7px solid Purple;margin:2px 0;"><strong>2021-12-13 19:48:34 EST - Shellene Cope</strong><sup style="float:right;">Additional comments</sup><div><span style="word-wrap:break-word;display:block;">Jeff Kelley has emailed the following reply: There has been no improvement in this issue. Have you heard anything from NESDIS? It's strange that the imagery on the STAR and NASA websites appear fine. <br> <br>https://weather.msfc.nasa.gov/cgi-bin/basicLooper.pl?category=goeseastabiconus&regex=10p35um&title=GOES-East%20ABI%20CONUS%20-%20%2010.35%20um%20-%20Band%2013&time_drop=show <br>https://www.star.nesdis.noaa.gov/GOES/conus_band.php?sat=G16&band=13&length=24 <br> <br>AWIPS and the FAA weather system continue to exhibit the same issue as this link: https://weather.cod.edu/satrad/ <br> <br>(SEC) </span></div></div></td></tr></tbody></table></div></div> <div><p>The following work notes were given about the Incident:</p><div style="margin-left:30px;"><table style="width:100%;"><tbody><tr><td><div style="padding:10px 15px; border:1px solid lightgray; border-left:7px solid gold;margin:2px 0;"><strong>2021-12-13 20:10:02 EST - Curtis Sims</strong><sup style="float:right;">Work notes</sup><div><span style="word-wrap:break-word;display:block;">@0054z - OMB received an update from Jeff Kelley <br>reply from: jeff.kelley@xxxxxxxx <br> <br>I just got off the phone with NCF. They are aware that data dropouts are <br>occurring (with all data types, not just satellite) after switching between <br>systems earlier today. The tech mentioned that it might be a firewall <br>issue. <br> <br>Jeff <br>(CS)</span></div></div></td></tr></tbody></table></div></div> <hr> <div> </div><div style="display:inline">Ref:MSG5533548</div></body></html>
python-users
archives: