quetz-mod_python-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Brunson <brun...@brunson.com>
Subject Re: Apache log messages
Date Sun, 10 Dec 2006 18:01:47 GMT
Jim Gallacher wrote:
> Eric Brunson wrote:
>> Platform: FreeBSD 6.1, Apache 2.0.55 (mpm_prefork), Python 2.4.3,
>> mod_python 3.3 (20061130232253 snapshot).
>>
>> Well, I see the problem is that httpd is getting started up close enough
>> to the same time every boot cycle that the worker processes are getting
>> the same semaphore file names.  For some reason FreeBSD doesn't clean
>> out /tmp on boot by default.  I can't seem to find any cleanup mechanism
>> for the files on shutdown, but the 80 or so mutex files seem to
>> correspond to server crashes 
>
> Yep. As a rule that is were the mutexes get consumed. I'm sure I don't
> need to tell you the ultimate solution is to figure out why apache is
> crashing and make it stop. :)

It's actually the server server that's crashing, not the apache server. 
We're trying to get some new hardware on deck to make sure it's not
physical before we do too much digging into FreeBSD.  We kind of had it
foisted upon is (more secure, blah, blah, blah, better networking, blah,
blah, you know the arguments), but no one here actually knows as much
about FreeBSD as the rest of us know about Linux, so we end up
scratching our heads and staring a lot when it comes to some of these
problems.

>
> My notes on this issue follow. They apply to Linux and may not be of
> any use on FreeBSD.

I've definitely hit that one on Linux before.  In fact, I just had a
support call for one of our linux boxes and that's the first thing I
thought of.  It turned out to be a missing log directory, but still.  I
always have to Google for that ipcs fix below, I think I'll just store
your email away for future reference.  :-)

> -------------------------------------------------------------------------
>
> Under some conditions Apache may crap out and the semaphores it
> reserved for the mod_python mutexes may not have been released. This
> may happen if it does not cleanly shutdown due to a hung mod_python
> instance.
>
> Each time apache is restarted it will grab additional semaphores until
> the server is starved for this resource.
>
> Check the number of semaphores available:
>
> # sysctl kernel.sem
> kernel.sem = 250        32000   32      128
>
>
> Get information about the semaphores
> # ipcs -s
>
> ------ Semaphore Arrays --------
> key        semid      owner      perms      nsems
> 0x0052e2c1 0          postgres  600        17
> 0x0052e2c2 32769      postgres  600        17
> 0x0052e2c3 65538      postgres  600        17
> 0x0052e2c4 98307      postgres  600        17
> 0x0052e2c5 131076     postgres  600        17
> 0x0052e2c6 163845     postgres  600        17
> 0x0052e2c7 196614     postgres  600        17
> 0x00000000 3145735    www-data  600        1
> 0x00000000 3178504    www-data  600        1
> 0x00000000 3211273    www-data  600        1
> 0x00000000 3244042    www-data  600        1
> 0x00000000 3276811    www-data  600        1
> 0x00000000 3309580    www-data  600        1
> 0x00000000 3342349    www-data  600        1
> 0x00000000 3375118    www-data  600        1
> 0x00000000 3407887    www-data  600        1
> 0x00000000 3440656    www-data  600        1
>
>
> The solution is to stop apache and clean out the dead semaphores,
> substituting the the correct apache user name for apache-user. Make
> sure apache is stopped before running this - we only want to remove
> the dead semaphores. On debian apache is run as the "www-data" user.
>
> ~localhost# ipcs -s | awk '/www-data/ { system("ipcrm sem " $2) }'
>
> Another one liner using perl to do the cleanup:
>
> ~localhost# ipcs -s | perl -ane '/^0x00000000/ && `ipcrm -s $F[1]`'
>
> Jim


Mime
View raw message