NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Have you tried Matts command on this data? On Jan 28, 2021, at 2:35 PM, McIDAS Help Desk <mug@xxxxxxxxxxxxx<mailto:mug@xxxxxxxxxxxxx>> wrote: His 1080 files from his NOAA S3 source are in: /nas/home/jayh/mcidas/data/doug/*.nc -Jay On 1/28/21 1:42 PM, Russ Dengel wrote: Jay, There are a few things that could be happening. The files vary in length could indicate that Matts files are including extra fields per record The Point ADDE servers do include Fortran code which could involve static arrays that are being exceeded There might be resource issues/differences between Matts machine and yours. Do we have some of Matts files? Russ On Jan 28, 2021, at 1:36 PM, McIDAS Help Desk <mug@xxxxxxxxxxxxx<mailto:mug@xxxxxxxxxxxxx>> wrote: Hi Russ- I've attached Doug's trce file. I don't know if he's tried SSEC servers, I'll ask as I think they do use our data at times. He has gotten these commands to work when he changes the INC= from INC=300 300 to INC=60 60 and does each hour individually. So, that seems to disprove the idea that there is a corrupt file or reading. I was wondering if the server is timing out or maybe it's something particular in his files. When he gets the error, it plots some data before failing. But I'm at a loss as to why I can't recreate the error with his files. Thanks, Jay On 1/28/21 10:30 AM, Russ Dengel wrote: Jay, The error is occurring in the ADDE Client/Server interface. I would need a trce file generated from the GLMN server running on Matt’s server to diagnose the problem past this point. Has Matt tried to use our (SSEC) server to try the same command? This —> "Our files look similar, but Doug's are all 15000 bytes larger.” is suspicious. Has Matt used GLMDISP to plot other data from these NOAA S3 datasets? Russ On Jan 27, 2021, at 7:23 PM, McIDAS Help Desk <mug@xxxxxxxxxxxxx<mailto:mug@xxxxxxxxxxxxx>> wrote: Hi Russ- I've been working with Doug Spangenberg from NASA Langley on a GLMDISP problem he is having. I'm at a loss. He has 1080 files, representing 6 hours of data. We've simplified the example to displaying over a WORL map, all files in one directory, and he sent me all his files. DSSERVE ADD DOUG2/GLM-GROUP GLMN TYPE=POINT INFO=/home/mcidas/data/GLM_GROUP.cfg DIRFILE=/nas/home/jayh/mcidas/data/doug/*.nc MAKFRM 1 800 1000 SF new frame GLMDISP WORL X DATASET=DOUG2/GLM-GROUP DAY=2019066 TIME=21:12 INC=300 300 TYPE=GROUP POS=GROUND COLOR=5 TOP='+ G16 LIGHTNING' 5 788 720 8 DEV=CCC TRACE=1 ECHO=YES This GLMDISP command takes a while for me, but does work. Doug's plots some data and fails with "PTDISP: Unable to read data record from server, PTDISP: Failed during record fetch." The last few lines of Doug's DEV=CCC show this: PTDISP* GOTO NVINI PTDISP* BACK FROM NVINI PTDISP* Read of server has failed. Wanted 4 PTDISP* Did PTDISP: Unable to read data record from server PTDISP: Failed during record fetch PTDISP* PTDISP* Number of records received = 1367248 PTDISP* Number of records plotted = 1367248 PTDISP* PTDISP* m0pttot totbytsent: 10938008 PTDISP - done GLMDISP - done It looks like Doug's gets about 1/3 done then is failing. Here is my DEV=CCC: PTDISP* GOTO NVINI PTDISP* BACK FROM NVINI PTDISP* PTDISP* Number of records received = 4168105 PTDISP* Number of records plotted = 4168105 PTDISP* PTDISP* m0pttot totbytsent: 33344864 PTDISP - done GLMDISP - done I'm out of ideas on what to suggest. He's using McIDAS-X 2020.1 on a Linux 7 machine, and I'm testing on the same. Our files look similar, but Doug's are all 15000 bytes larger. He got them from a NOAA S3 archive, I got mine from /arcdata. There are subtle differences in the header when checking with ncdump -h, but they didn't seem relevant. What does the "Unable to read data record from server" error mean? And do you have any other thoughts for Doug? Thanks, Jay -------- Forwarded Message -------- Subject: Re: [EXTERNAL] Re: G16 GLM Date: Tue, 26 Jan 2021 23:54:07 +0000 From: Spangenberg, Douglas A. (LARC-E302)[Science Systems & Applications, Inc.] <douglas.a.spangenberg@xxxxxxxx><mailto:douglas.a.spangenberg@xxxxxxxx> To: McIDAS Help Desk <mug@xxxxxxxxxxxxx><mailto:mug@xxxxxxxxxxxxx> Hi Jay, I have 1080 files as well. The files I’ve been using were downloaded from a NOAA S3 archive. I also tried running the command with INC=60 60 5X through day 066, hours 16-21, and it worked for all the files in the dsserve command DIRFILE= below. It looks like the INC=300 300 is the part that didn’t work for me with unable to read from server error. I’m attaching the log of my run with DEV= ECHO= added. I also put a copy of the files (tar file) I’m using in this location if you wanted to download and try: https://satcorps.larc.nasa.gov/prod/NOAA-G16-GLM/2019/066/ DIRFILE=/gsaas_fsx/SatCORPS-Ancilliary/NOAA-G16-GLM/2019/066/*/*.nc Thanks, Doug From: McIDAS Help Desk <mug@xxxxxxxxxxxxx><mailto:mug@xxxxxxxxxxxxx> Date: Tuesday, January 26, 2021 at 2:35 PM To: "Spangenberg, Douglas A. (LARC-E302)[Science Systems & Applications, Inc.]"<douglas.a.spangenberg@xxxxxxxx><mailto:douglas.a.spangenberg@xxxxxxxx> Subject: Re: [EXTERNAL] Re: G16 GLM Hi Doug- Thanks for working through this with me. I think I'm working with the same dataset as you in the same directory structure now. I obtained my files from SDS and have 180 files in each directory named /home/jayh/mcidas/data/glm/2019/066/16/*, ../2019/066/17/*, through ../2019/066/21/*. So, it's searching through 1080 files, which doesn't seem like too many. Do you have the same file count? Can you add DEV=CCC and ECHO=YES to your last glmdisp.k over the WORL map and send me the output? My command is receiving and plotting 4168274 records with no errors. I'm using a DIRFILE= with /home/jayh/mcidas/data/glm/2019/066/*/*.nc Also, did you get your files from CLASS, SDS or another source? Thanks, Jay On 1/22/21 8:01 PM, Spangenberg, Douglas A. (LARC-E302)[Science Systems & Applications, Inc.] wrote: Hi Jay, \A0 I ran it with WORL map (see below) and it still gave the error about \201C Unable to read data record from server\201D .\A0 However, it did make a map with lightning displayed although likely less than normal depending on what file the program stopped on.\A0 I attached the trace log. Perhaps it would be good to add to the program more information about the file it was parsing, the date and time of the file when the message happens since there are so many files, it can be hard to track.\A0 I did see the files were all same permissions and similar sizes so at least that was consistent. \A0 For individual hours in the dsserve command, it worked on all of them without a read server error, so there might be a problem going across the hours or for more than a certain number of files?\A0 I also put all hours in one directory and used no * in the path but that didn\2019 t work either. \A0 glmdisp.k WORL 1 DATASET=G16/GLM-GROUP2 DAY=2019066 TIME=21:12 INC=300 300 TYPE=GROUP POS=GROUND COLOR=5 TRACE=1 TOP='+ G16 LIGHTNING' 5 488 520 8 \A0 Thanks, Doug \A0 \A0 From: McIDAS Help Desk <mug@xxxxxxxxxxxxx><mailto:mug@xxxxxxxxxxxxx> Date: Thursday, January 21, 2021 at 5:39 PM To: "Spangenberg, Douglas A. (LARC-E302)[Science Systems & Applications, Inc.]"<douglas.a.spangenberg@xxxxxxxx><mailto:douglas.a.spangenberg@xxxxxxxx> Subject: Re: [EXTERNAL] Re: G16 GLM \A0 Hi Doug- I put my GLM files in a similar directory structure.\A0 However, I only have 16-21Z for day 2019064 and 2019064.\A0 I've only been able to replicate the No NetCDF file data satisfies error.\A0 A guess would be there is a corrupt file or non-GLM file getting lumped in when searching which might lead to the Unable to read data record error. Thoughts: 1.\A0 Instead of overlaying over the satellite image, what happens if you use MAP WORL for the domain (with the command that gives the Unable to read data record error)? 2.\A0 For the command that gives the Unable to read data record error, can you add TRACE=1 to the command?\A0 Then a "trce" file should be written into your working directory.\A0 Please send me that file. Thanks, Jay On 1/20/21 10:41 AM, Spangenberg, Douglas A. (LARC-E302)[Science Systems & Applications, Inc.] wrote: Hi Jay, \A0 I\2019 m getting the PTDISP: No NetCDF file data satisfies the request when I use no * in the directory path and run the same command one hour at a time.\A0 When I put one star in the path for the hour on day 066, I get the other message PTDISP: Unable to read data record from server, PTDISP: Failed during record fetch. Also, on day 2019066, I have GLM data for hours 16-21 only in there, not the entire day. \A0 Thanks, \A0 Doug \A0 \A0 From: McIDAS Help Desk <mug@xxxxxxxxxxxxx><mailto:mug@xxxxxxxxxxxxx> Date: Friday, January 15, 2021 at 1:42 PM To: "Spangenberg, Douglas A. (LARC-E302)[Science Systems & Applications, Inc.]" <douglas.a.spangenberg@xxxxxxxx><mailto:douglas.a.spangenberg@xxxxxxxx> Subject: Re: [EXTERNAL] Re: G16 GLM \A0 Hi Doug- With the data from 2019066 displayed over that domain, I'm getting the "PTDISP: No NetCDF file data satisfies the request" message, which makes sense since there is no lightning data in the frame. I'm still unsure why you are getting the "PTDISP: Unable to read data record from server" message. Could you try defining your DIRFILE= without the "*" wildcards (*.nc is fine, just change the others to point to the 2019066 data directly)?\A0 I'm curious to see what error you get. Thanks, Jay On 1/14/21 6:10 PM, Spangenberg, Douglas A. (LARC-E302)[Science Systems & Applications, Inc.] wrote: Hi Jay, \A0 1. From the dsserve command: G16/GLM-GROUP\A0 POINT GLMN Info=/home/mcidas/data/GLM_GROUP.cfg DIRFILE=/gsaas_fsx/SatCORPS-Ancilliary/NOAA-G16-GLM/2019/0*/*/*.nc It\2019 s local and not on a remote server. \A0 2.\A0 It\2019 s from McIDAS-X 2020.1 version with Linux 7.9 \A0 3. Navigation is from these commands with 500x700 frame size imgcopy.k AGOES16/CONUS A.7000 SIZE=SAME BAND=13 MAG=-1 DAY=2019066 TIME=18:22 18:26 imgremap.k A.7000 A.7001 PRO=RECT LATLON=41.4 91.4 RES=1.3 SIZE=500 700 imgdisp.k A.7001 2 LATLON=41.4 91.4 \A0 MAG=-1 \A0 thanks, \A0 Doug \A0 \A0 From: McIDAS Help Desk <mug@xxxxxxxxxxxxx><mailto:mug@xxxxxxxxxxxxx> Date: Thursday, January 14, 2021 at 1:27 PM To: "Spangenberg, Douglas A. (LARC-E302)[Science Systems & Applications, Inc.]" <douglas.a.spangenberg@xxxxxxxx><mailto:douglas.a.spangenberg@xxxxxxxx> Subject: [EXTERNAL] Re: G16 GLM \A0 Hi Doug- Regarding the 2nd command, you are correct, that is the error received when there is no lightning in the frame. For the 1st command, I downloaded the 2019066 GOES 16 GLM files and put them in a local directory to do some testing.\A0 I wasn't able to recreate the error.\A0 My GLMDISP completed with no errors and showed some lightning data over Colorado and Utah.\A0 So, I have a few questions: 1. I noted the error you are getting is the same when there is a period "." somewhere in the DIRFILE= (inq. 17059<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmcidas.ssec.wisc.edu%2Finquiry-x%2Findex.php%3Finquiry%3D17059&data=04%7C01%7Cdouglas.a.spangenberg%40nasa.gov%7Cba0080e19c07490fcdde08d8c23188a2%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637472865286472700%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=A6SRxoyMcRjvahIT8oI7muPFSsE7bIRNvvJrHOE5feg%3D&reserved=0> -\A0 which hasn't been released yet).\A0 I was curious to see what the DSSERVE command was for the G16/GLM-GROUP dataset (and whether it's local or remote)?\A0 However, I would expect your other GLMDISP command to fail the same way, so I'm not confident this is the answer. 2. If #1 didn't solve the problem, what version of -X are you using, what operating system? 3. What is the IMGDISP command you are using as navigation, and what size frame are you using? Thanks, Jay On 1/13/21 5:33 PM, Spangenberg, Douglas A. (LARC-E302)[Science Systems & Applications, Inc.] wrote: McIDAS Help, \A0 I\2019 ve been running some glmdisp commands with historical GOES-16 lightning netcdf files and have seen the following messages.\A0 Not on every time period just for some. Do you know what is happening? \A0 If I run this as separate commands with 20 minutes duration of lightning for each, it doesn\2019 t give any error messages: glmdisp.k X 2 DATASET=G16/GLM-GROUP DAY=2019066 TIME=21:12 INC=300 300 TYPE=GROUP NAV=2 POS=GROUND COLOR=5 TOP='+ G16 LIGHTNING' 5 488 520 8 Plotting GLM data from G16/GLM-GROUP.ALL PTDISP: Unable to read data record from server : Failed during record fetch\A0 \A0 \A0 \A0 \A0 \A0 \A0 \A0 \A0 I also get this message. I\2019 m assuming for those periods when no lightning occurred in the frame: glmdisp.k X 2 DATASET=G16/GLM-GROUP DAY=2019064 TIME=16:32 INC=280 280 TYPE=GROUP NAV=2 POS=GROUND COLOR=5 TOP='+ G16 LIGHTNING' 5 488 520 8 Plotting GLM data from G16/GLM-GROUP.ALL PTDISP: No NetCDF file data satisfies the request GLMDISP - done \A0 All glmdisp commands were run with G16 IR images on frames showing the upper Midwest USA. \A0 Thanks, \A0 Doug --> --> --> --><g16glm.2019066.output.log> <trce_doug.txt>
mcidas-x archives: