NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Daryl,
No each process creates a temporary working subdirectory to run in based
on the process id.
Greg
On Apr 30, 2012, at 1:23 PM, daryl herzmann wrote:
> Greg,
>
> Are all these processes running out of the same CWD (directory)? Try
> creating temp directories for each process and run the code from those
> directories.
>
> daryl
>
> On Mon, 30 Apr 2012, Greg Stossmeister wrote:
>
>> Daryl,
>> I have several shell scripts that I'm running out of cron every 5 minutes.
>> Each shell script runs 10 gpmap_gf processes in sequence. I've tried running
>> 1 - 6 scripts at a time. This typically works fine during the day with one
>> of the these scripts completing in about 2 minutes. As evening comes on they
>> take longer and longer to run and it seems like that take more and more
>> memory. From the "top" command the scripts often use 500-800 MB of memory
>> but in the evening this seems to mushroom to > 3GB per script. The load on
>> the machine at night from these scripts alone jumps to >30 and by morning
>> the machine usually dies with out of memory errors even though I'm
>> automatically killing the scripts when they run longer than 2 minutes.
>>
>> Looking at /var/log/debug.log I'm seeing segfault errors:
>>
>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2164]: segfault at 0 ip
>> 000000392692ff7f sp 00007fff8d981128 error 4 in
>> libc-2.12.so[3926800000+186000]
>> Apr 26 17:07:25 sferic abrt[2179]: saved core dump of pid 2164
>> (/export/ldm/home/gempak/GEMPAK6.4.0/os/linux64/bin/gpmap_gf) to
>> /var/spool/abrt/ccpp-201
>> 2-04-26-17:07:25-2164.new/coredump (827392 bytes)
>> Apr 26 17:07:25 sferic abrtd: Directory 'ccpp-2012-04-26-17:07:25-2164'
>> creation detected
>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2239]: segfault at 0 ip
>> 000000392692ff7f sp 00007ffffd173658 error 4 in
>> libc-2.12.so[3926800000+186000]
>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2242]: segfault at 0 ip
>> 000000392692ff7f sp 00007fff8f4df6f8 error 4 in
>> libc-2.12.so[3926800000+186000]
>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2247]: segfault at 0 ip
>> 000000392692ff7f sp 00007fff73574d18 error 4 in
>> libc-2.12.so[3926800000+186000]
>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2261]: segfault at 0 ip
>> 000000392692ff7f sp 00007fff8bda1358 error 4 in
>> libc-2.12.so[3926800000+186000]
>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2245]: segfault at 0 ip
>> 000000392692ff7f sp 00007fff71495a28 error 4 in
>> libc-2.12.so[3926800000+186000]
>> Apr 26 17:07:25 sferic kernel: Pid 2245(gpmap_gf) over core_pipe_limit
>> Apr 26 17:07:25 sferic kernel: Skipping core dump
>> Apr 26 17:07:25 sferic abrt[2260]: not dumping repeating crash in
>> '/export/ldm/home/gempak/GEMPAK6.4.0/os/linux64/bin/gpmap_gf'
>> Apr 26 17:07:25 sferic abrt[2279]: not dumping repeating crash in
>> '/export/ldm/home/gempak/GEMPAK6.4.0/os/linux64/bin/gpmap_gf'
>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2289]: segfault at 0 ip
>> 000000392692ff7f sp 00007fffca7118a8 error 4 in
>> libc-2.12.so[3926800000+186000]
>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2286]: segfault at 0 ip
>> 000000392692ff7f sp 00007fffef00ac98 error 4 in
>> libc-2.12.so[3926800000+186000]
>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2303]: segfault at 0 ip
>> 000000392692ff7f sp 00007fff92019618 error 4 in
>> libc-2.12.so[3926800000+186000]
>> Apr 26 17:07:25 sferic kernel: Pid 2303(gpmap_gf) over core_pipe_limit
>>
>> Greg
>>
>> On Apr 30, 2012, at 12:10 PM, daryl herzmann wrote:
>>
>>> On Mon, 30 Apr 2012, Greg Stossmeister wrote:
>>>
>>>> Does anyone generate a lot of individual NEXRAD level III products with
>>>> gpmap_gf? I'm trying to generate real-time plots of NOQ Reflectivity and
>>>> NOU Velocity from 30 radars in the midwest and its crashing my server
>>>> after a few hours, even when I only run 3 plots at a time. I'm running
>>>> GEMPAK6.4.0 on a RHEL 6 machine with 64 GB of memory. I'm wondering what
>>>> I'm doing wrong and if someone has a better way of doing this.
>>>
>>> crashing your server, how? Exhausting memory? kernic panic? Are the
>>> processes not going away once running them? How are you running them?
>>>
>>> daryl
>>>
>>> --
>>> /**
>>> * Daryl Herzmann
>>> * Assistant Scientist -- Iowa Environmental Mesonet
>>> * http://mesonet.agron.iastate.edu
>>> */
>>
>> -----------------------------------------------------------
>> Greg Stossmeister
>> Head - Collaborative Tools and Metadata Group
>> NCAR/Earth Observing Laboratory (EOL)
>> Telephone: (303)497-8692 Facsimilie:(303)497-2044
>> e-mail: gstoss@xxxxxxxx
>> Mailing Address: Shipping Address:
>> NCAR/EOL UCAR Foothills Lab 1
>> P.O. Box 3000 3450 Mitchell Ln
>> Boulder, CO 80307 Boulder, CO 80301
>> ------------------------------------------------------------
>>
>>
>>
>>
>>
>> _______________________________________________
>> gembud mailing list
>> gembud@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>>
>
> --
> /**
> * Daryl Herzmann
> * Assistant Scientist -- Iowa Environmental Mesonet
> * http://mesonet.agron.iastate.edu
> */
-----------------------------------------------------------
Greg Stossmeister
Head - Collaborative Tools and Metadata Group
NCAR/Earth Observing Laboratory (EOL)
Telephone: (303)497-8692 Facsimilie:(303)497-2044
e-mail: gstoss@xxxxxxxx
Mailing Address: Shipping Address:
NCAR/EOL UCAR Foothills Lab 1
P.O. Box 3000 3450 Mitchell Ln
Boulder, CO 80307 Boulder, CO 80301
------------------------------------------------------------
gembud archives: