Home » Blogs » sandip's blog

Auto restart apache on segmentation fault error

Fri, 11/16/2007 - 16:36 — sandip

Recently a particular folder in a site started coming up with blank php pages and the root cause being Segmentation fault after memory exhaustion. The subfolder contents would only come up after doing a restart of apache.

So something tried to access a region of memory that it did not have rights to. May be due to either bad RAM, or a code problem.

My first instinct was eAccelerator as I had done a recent php update and had not re-compiled it.

I still had the same issue the next day and bumped up the memory_limit from 16M to 32M in php.ini and from 32M to 64M in eaccelerator.ini .

The issue has not come up again (fingers crossed). But, to save on the downtime and having to manually restart apache, I put in a cron script to monitor the error log and restart apache automatically in case it happens again.

#!/bin/bash

sfile=/tmp/.seg_fault
TO_MAIL=user@domain.tld

if [ -f "${sfile}" ]; then
        old_count=`cat /tmp/.seg_fault`
else
        old_count=0
fi

new_count=`/bin/grep -c "Segmentation fault (11)" /var/log/httpd/error_log`
echo $new_count > $sfile

if [ "${new_count}" -ne "0" ] && [ "${old_count}" -ne "${new_count}" ]; then
        /sbin/service httpd restart
        echo "segmentation fault detected and httpd restarted" | mail -s "httpd restarted" $TO_MAIL
fi

exit 0

Bookmark/Search this post with
  • del.icio.us
  • Digg
  • Facebook
  • identi.ca
  • LinkedIn
  • SlashDot
  • StumbleUpon
  • Technorati
  • Twitter

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Mon, 11/19/2007 - 12:29 — Simon (not verified) Did increasing the memory limits fix your problem?

I've been running into this problem as well, and I'm thinking of adding your script to my crontab. But I would like to know if upping the memory limits has kept the problem from recurring for you?

Tue, 11/20/2007 - 12:27 — sandip Yes, it seems to have

Also to check, is that the eAccelerator cache directory is writable by apache?

Turn on syslog logging and keep a look out on errors and memory exhaustion reported.

Fri, 11/16/2007 - 16:52 — sandip report php errors

To further enable web developers, notification of errors being logged to the syslog, the below script was put in cron and runs daily at 11:59pm.

#!/bin/bash

TO_EMAIL=user1@domain.tld />CC_EMAIL=user2@domain.tld />OUT_FILE=`/bin/mktemp /tmp/.httpd_error.XXXX`
DATE=$(/bin/date "+%b %_d")

/bin/grep "$DATE" /var/log/messages| /bin/grep httpd > $OUT_FILE

if [ -s "$OUT_FILE" ]; then
        /bin/cat $OUT_FILE | /bin/mail -s "httpd error report" -c $CC_EMAIL $TO_EMAIL
fi

/bin/rm -f $OUT_FILE

exit 0

php.ini settings:

error_reporting  =  E_ALL & ~E_NOTICE
display_errors = Off
log_errors = On
ignore_repeated_errors = On
report_memleaks = On
error_log = syslog

Comment