"core catcher" for Linux

Your software is crashing and all you get is a lousy core file - wouldn't it be great if they system could send you an email containing a human-readable backtrace instead?

Fear not, a little contraption made up of incron and gdb can do this for you. In my example I'll start dovecot (which has been compiled with debugging enabled!) from runit (since I want to keep it running after it crashed). The /etc/service/dovecot/run file im using for that purpose looks like this:

echo '/var/core/%p' > /proc/sys/kernel/core_pattern
ulimit -n 8096
ulimit -c unlimited
cd /tmp
exec /usr/local/sbin/dovecot -F

This tells the kernel to put the core files into /var/core/$PROCESSID. And by using ulimit -c unlimited I make sure there are corefiles at all.

So, if a new file core file arrives in that directory, we should create a backtrace and send it via email. For this, we use incron by editing the incrontab:

/var/core IN_CLOSE_WRITE /usr/local/scripts/create_bt $@/$#

And last but not least /usr/local/scripts/create_bt does the actual work:

echo $1
binary=`file $1 | awk -F\' '{print $2}' | awk '{print $1}'`
echo $binary
gdb -x ~root/bt /usr/local/dovecot/libexec/$binary $1 | mailx -s Backtrace:$binary r@sys4.de 2>/dev/null
#rm -f $1

gdb executes the commands from ~root/bt which simply contains:

bt full
Ralf Hildebrandt, 13 Apr 2013