Bycrom

Unassisted GDB Sessions

December 13th, 2008 by Tastic

Bugs can be hard to find, but I guess if it were easy, I might be out of a job. Nah…. they still need someone to fix those bugs. However, the point being that because dev machines are tainted, sometimes you can’t find the bugs, sometimes it’s because you never take the right path in the software to cause bugs, and almost every time no one can tell you anything relevant about how they ran into the bugs. Ok, the last part might not be fair.

We’ve found a large number of times that we’ve had to resort to running a debugger, gdb in our case, and a debug build of the software in question on a client box to find errors. Usually, this sucks as a developer has to babysit the software while waiting for the user to trigger something interesting. Recently, I’ve found myself in a situation where I need to do this for multiple machines and I’m severely lacking the patience to follow that. So like all lazy programmers, I’ve been obsessed with finding a way to take myself out of the loop.

The solution was simple once I got it right. Set gdb to launch in batch mode, set some interesting breakpoints, and a loop to backtrace and continue until quit. Get the log file at the end of a few days of testing. Getting that to actually work was a little rough though. 

First off, I knew I needed a separate gdbinit file than my normal one and a easy way to launch. I’m in Windows here, so a shortcut could easily handle the launching. A shortcut like the one below works well.

“C:\Program Files\Your Debug Software\gdb.exe” –batch -nx –command=unassisted.gdbinit

This assumes you’ve bundled gdb, that you don’t want it to attempt to use the normal .gdbinit, and you’ve created a unassisted.gdbinit. The “–batch” option will prevent gdb from prompting the user, and will automatically quit when the command file ends I then named the shortcut something like “Unassisted Debug Session”

Now, to the guts of the gdb batch file:
# Logs will append to gdb.txt in the working directory by default
set logging on
file YourApp.exe

# Settings good for logging in this case
set height 0
set new-console off

# Seems nicer in Windows.
set mapexceptionstosignals off
set debugevents off
set debugexceptions off

# Set first break and run
break main
run

# For DLLs, I've had problems setting breakpoints before the app actually starts.
break error_log_function

# I program ObjC in windows... yeah, that's weird
break '-[NSException raise]'
break '-[DatabaseConnection postError:]'

# autorelease with no pool in my copy of Foundation framework
break NSAutoreleasePool.m:250

# HERE! This is were the real fun is - $_exitcode gets set in gdb when your app exits
set $_exitcode = 9999

# Corrupt stacks frames break the batch command loop or user-defined function, so continues happen separately from the backtrace. We'll do the backtrace at all stop on a hook instead, as long as the program has not quit
define hook-stop
  if ($_exitcode == 9999)
    backtrace full
  end
end

# The continue loop - this will occur until the program exits
while ($_exitcode == 9999)
  continue
end

quit


Now, just run on multiple machine and grab the logs afterward. Hopefully, the backtrace full will be enough to track down your bugs. Happy Hacking.

Stacks & Segfaults