Auto restart apache on segmentation fault error

Recently a particular folder in a site started coming up with blank php pages and the root cause being Segmentation fault after memory exhaustion. The subfolder contents would only come up after doing a restart of apache.

So something tried to access a region of memory that it did not have rights to. May be due to either bad RAM, or a code problem.

My first instinct was eAccelerator as I had done a recent php update and had not re-compiled it.

I still had the same issue the next day and bumped up the memory_limit from 16M to 32M in php.ini and from 32M to 64M in eaccelerator.ini .

The issue has not come up again (fingers crossed). But, to save on the downtime and having to manually restart apache, I put in a cron script to monitor the error log and restart apache automatically in case it happens again.

#!/bin/bash

sfile=/tmp/.seg_fault
TO_MAIL=user@domain.tld

if [ -f "${sfile}" ]; then
        old_count=`cat /tmp/.seg_fault`
else
        old_count=0
fi

new_count=`/bin/grep -c "Segmentation fault (11)" /var/log/httpd/error_log`
echo $new_count > $sfile

if [ "${new_count}" -ne "0" ] && [ "${old_count}" -ne "${new_count}" ]; then
        /sbin/service httpd restart
        echo "segmentation fault detected and httpd restarted" | mail -s "httpd restarted" $TO_MAIL
fi

exit 0

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Did increasing the memory limits fix your problem?

I've been running into this problem as well, and I'm thinking of adding your script to my crontab. But I would like to know if upping the memory limits has kept the problem from recurring for you?

Yes, it seems to have

Also to check, is that the eAccelerator cache directory is writable by apache?

Turn on syslog logging and keep a look out on errors and memory exhaustion reported.

report php errors

To further enable web developers, notification of errors being logged to the syslog, the below script was put in cron and runs daily at 11:59pm.

#!/bin/bash

TO_EMAIL=user1@domain.tld />CC_EMAIL=user2@domain.tld />OUT_FILE=`/bin/mktemp /tmp/.httpd_error.XXXX`
DATE=$(/bin/date "+%b %_d")

/bin/grep "$DATE" /var/log/messages| /bin/grep httpd > $OUT_FILE

if [ -s "$OUT_FILE" ]; then
        /bin/cat $OUT_FILE | /bin/mail -s "httpd error report" -c $CC_EMAIL $TO_EMAIL
fi

/bin/rm -f $OUT_FILE

exit 0

php.ini settings:

error_reporting  =  E_ALL & ~E_NOTICE
display_errors = Off
log_errors = On
ignore_repeated_errors = On
report_memleaks = On
error_log = syslog

Comment