Tip: Poor man’s qemu breakpoint

I’ve written before about how you can use qemu + gdb to debug a guest. Today I was wondering how I was going to debug a problem in a BIOS option ROM, when Stefan Hajnoczi mentioned this tip: Insert

1: jmp 1b

into the code as a “poor man’s breakpoint”. In case you don’t know what that assembly code does, it causes a jump back (b) to the previous 1 label. In other words, an infinite loop.

After inserting that into the option ROM, recompiling and rebooting the virtual machine, it hangs in the boot, and hitting ^C in gdb gets me straight to the place where I inserted the loop.

(gdb) target remote localhost:1234
Remote debugging using localhost:1234
0x0000fff0 in ?? ()
(gdb) set architecture i8086
The target architecture is assumed to be i8086
(gdb) cont
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x00000045 in ?? ()
(gdb) info registers
eax            0xc100	49408
ecx            0x0	0
edx            0x0	0
ebx            0x0	0
esp            0x6f30	0x6f30
ebp            0x6f30	0x6f30
esi            0x0	0
edi            0x0	0
eip            0x45	0x45
eflags         0x2	[ ]
cs             0xc100	49408
ss             0x0	0
ds             0xc100	49408
es             0x0	0
fs             0x0	0
gs             0x0	0
(gdb) disassemble 0xc1000,0xc1050
Dump of assembler code from 0xc1000 to 0xc1050:
...
   0x000c103c:	mov    %cs,%ax
   0x000c103e:	mov    %ax,%ds
   0x000c1040:	mov    %esp,%ebp
   0x000c1043:	cli    
   0x000c1044:	cld    
   0x000c1045:	jmp    0xc1045
   0x000c1047:	jmp    0xc162c
   0x000c104a:	sub    $0x4,%esp
   0x000c104e:	mov    0xc(%esp),%eax
End of assembler dump.

Look, my infinite loop!

I can then jump over the loop and keep single stepping^*:

(gdb) set $eip=0x47
(gdb) si
0x0000062c in ?? ()
(gdb) si
0x0000062e in ?? ()
(gdb) si
0x00000632 in ?? ()

I did wonder if I could take Stefan’s idea further and insert an actual breakpoint (int $3) into the code, but that didn’t work for me.

Note to set breakpoints, the regular gdb break command doesn’t work. You have to use hardware-assisted breakpoints instead:

(gdb) hbreak *0xc164a
Hardware assisted breakpoint 1 at 0xc164a
(gdb) cont
Continuing.

Program received signal SIGTRAP, Trace/breakpoint trap.
0x0000064a in ?? ()

Note:
^* If you find that single stepping doesn’t work, make sure you are using qemu in TCG mode (-M accel=tcg), as KVM code apparently cannot be single-stepped.

Tip: Use gdbserver to debug qemu running under libguestfs

If qemu crashes or fails when run under libguestfs, it can be a bit hard to debug things. However a small qemu wrapper and gdbserver can help.

Create a file called qemu-wrapper chmod +x and containing:

#!/bin/bash -

if ! echo "$@" | grep -sqE -- '-help|-version|-device \?' ; then
  gdbserver="gdbserver :1234"
fi

exec $gdbserver /usr/bin/qemu-system-x86_64 "$@"

Set your environment variables so libguestfs will use the qemu wrapper instead of running qemu directly:

$ export LIBGUESTFS_BACKEND=direct
$ export LIBGUESTFS_HV=/path/to/qemu-wrapper

Now we run guestfish or another virt tool as normal:

$ guestfish -a /dev/null -v -x run

When qemu starts up, gdbserver will run and halt the process, printing:

Listening on port 1234

At this point you can connect gdb:

$ gdb
(gdb) file /usr/bin/qemu-system-x86_64
(gdb) target remote tcp::1234
set breakpoints etc here
(gdb) cont

Half-baked idea: Enable strace/gdb on just one subprocess

For more half-baked ideas, see the ideas tag.

Why’s it not possible to run a deeply nested set of programs (eg. a large build) and have strace or gdb just trigger on a particular program? For example you could do a regular expression match on a command line:

exectrace --run=strace --cmd="\./prog.*-l" -- make check

would (given this theoretical exectrace tool) trigger strace when any child process of the make check matches the regexp \./prog.*-l.

Or perhaps you could trigger on current directory:

exectrace --run=strace --cwd=tests -- make check

I guess this could be implemented using PTRACE_O_TRACEEXEC, and it should have less overhead then doing a full recursive strace, and less annoyance than trying to trace a child in gdb (which AFAIK is next to impossible).

Tip: gdb break if first function argument has a value

I wanted to find out who was closing stdout in my program (ie. who is calling close(1);).

This seems to work in gdb:

(gdb) break close if $rdi == 1

The explanation is that on x86-64, register %rdi is used to pass the first argument to a function. In gdb conditionals, register names are prefixed by $. Hence the condition checks if the first argument to the function is 1.

Tip: Debugging the early boot process with qemu and gdb

Update: A much easier way is to use gdbserver.

Start qemu with the following parameters:

$ qemu-system-x86_64 -s -S -m 512 -hda winxp.img

And connect with gdb like this:

$ gdb
(gdb) target remote localhost:1234
(gdb) set architecture i8086
(gdb) break *0x7c00
(gdb) cont

This will breakpoint at 0x7c00, which is when the boot sector has been loaded into memory by the BIOS and control is passed to the boot sector.

You can use ordinary gdb commands to disassemble and debug the guest.