Tag Archives: systemtap

Using systemtap to trace into libraries

Systemtap is able to trace into userspace libraries. It’s actually very simple:


global last;

function display_time () {
      now = gettimeofday_us ();
      delta = 0;
      if (last > 0)
            delta = now - last;
      last = now;

      printf ("%d (+%d):", now, delta);

probe begin {
      last = 0;
      printf ("ready\n");

probe process("/usr/lib*/libguestfs.so.0")
            .function("guestfs_[a-z]*") ? {
      printf ("\t%s %s\n", probefunc(), $$parms);

Run this script (as root) and in another terminal run any program that uses libguestfs, and you’ll see this output:

$ sudo /tmp/test.stap
1318252148697989 (+0):	guestfs_create 
1318252148698028 (+39):	guestfs_is_config g=0x123ab50
1318252149060531 (+362503):	guestfs_set_pgroup g=0x123ab50 pgroup=0x1
1318252149060588 (+57):	guestfs_add_drive g=0x123ab50 filename=0x123ad50
1318252149060648 (+60):	guestfs_safe_malloc g=0x123ab50 nbytes=0x8
1318252149060668 (+20):	guestfs_safe_realloc g=0x123ab50 ptr=0x123add0 nbytes=0x10
1318252149060689 (+21):	guestfs_safe_strdup g=0x123ab50 str=0x368567df1b
1318252149060705 (+16):	guestfs_safe_realloc g=0x123ab50 ptr=0x123add0 nbytes=0x18
1318252149060719 (+14):	guestfs_safe_strdup g=0x123ab50 str=0x7fffea8473f0
1318252149060736 (+17):	guestfs_is_config g=0x123ab50
1318252149060749 (+13):	guestfs_launch g=0x123ab50


Filed under Uncategorized

Half-baked ideas: feedback-directed fuzz testing of filesystems

For more half-baked ideas, see my ideas tag

Fuzz-testing (a.k.a random testing) is an automatic method of testing where you feed in random data and try to make the program crash.

You might, for example, write random bytes to a disk, then ask Linux to mount it. This is not very sophisticated since Linux is unlikely to find a filesystem signature in pure random bytes, so the test will almost always fail to test anything. A better approach is to take an existing disk image and write random bits to parts of it. This is what Steve Grubb’s fsfuzzer does, but it’s still not the state of the art, although Steve found several bugs with it.

Suppose we think of a filesystem as just an array of integers:

4, 5, 13, 0, 2, ...

The code in the kernel to read this filesystem might start off:

if ((fs[0] & 3) != 0) {
  printk ("invalid filesystem");
  return -EINVAL;

If you generate random filesystems, then only 1 in 4 filesystems will get past this first test in the code, so only 1 in 4 of your random tests is testing anything beyond the first statement.

A better approach is to use feedback from the kernel code to evolve your fuzz tests. You score each fuzz test based on how far into the kernel code it gets, then you use that score to evolve your tests using standard genetic algorithm techniques. The idea is that your fuzz tests evolve to test more and more of the kernel code you are interested in testing, instead of just randomly falling down at the first hurdle.

(This technique is well-known in research as feedback-directed fuzz testing, feedback-directed random testing, or evolutionary fuzz testing. As far as I can tell no one is using it on Linux.)

Here’s the half-baked idea: Use systemtap probes as a way to score the fuzz-tests.

We write a systemtap module which inserts probes at every possible point in the file we are interested in testing, say, fs/minix/*.c. These probes just print a simple “I am here” type message.

When we run our fuzz test, we score it according to how much output it produces (by measuring “dmesg” before and after the test). Hitting probe points scores 1. Causing a kernel oops scores a lot more. You’d probably want to do this in a VM, rather than on your host kernel …

Starting with real, but small filesystems, we randomly fuzz them to generate our initial test cases, then evolve those according to how much they score. The test cases are individually tested using the same method as Steve’s fsfuzzer — ie. mounting the filesystem and running through a few simple system calls (read the directory, read files, read extended attrs ..)

Test cases which hit the equivalent of the 1-in-4 problem above are quickly evolved out, and the test cases we are left with hopefully explore and test more of the code.

Leave a comment

Filed under Uncategorized

Tip: use SystemTap to monitor SELinux changes to files

Something unknown was changing the labels on certain devices behind my back. We couldn’t find out what it was using ordinary diagnostics, so I decided to investigate if we could do this with SystemTap. I quickly found an existing script to monitor changes in ordinary file attributes. This won’t work for SELinux labels though because those are stored in ext2/3/4 extended attributes (xattrs).

Basically I had to modify that script to monitor calls to setxattr instead.

Using the LXR I found that the call is implemented in Linux in fs/xattr.c, function vfs_setxattr. I had to modify the script to probe that kernel function, and the parameters are slightly different too.

I also had to install the correct kernel-{,PAE-}debuginfo package corresponding to my installed kernel. This is how SystemTap is able to resolve symbols in the current kernel.

/* Watch changes to xattrs on an inode.
 * https://rwmj.wordpress.com/2010/03/09/tip-use-systemtap-to-monitor-selinux-changes-to-files/

probe kernel.function("vfs_setxattr")
  dev_nr = $dentry->d_inode->i_sb->s_dev
  inode_nr = $dentry->d_inode->i_ino

  if (inode_nr == $1)
    printf ("%s(%d) %s 0x%x/%u %d %s %s\n",
      execname(), pid(), probefunc(), dev_nr, inode_nr, uid(),
      kernel_string ($name), kernel_string_n ($value, $size))

Then run it with:

# stap -v /tmp/inodewatchxattr.stp inodenum

The bug turned out to be udevd, which I don’t think anyone was expecting …

libvirtd(4338) vfs_setxattr 0x5/166267 0 security.selinux system_u:object_r:svirt_image_t:s0:c177,c272
udevd(28299) vfs_setxattr 0x5/166267 0 security.selinux system_u:object_r:fixed_disk_device_t:s0
udevd(28299) vfs_setxattr 0x5/166267 0 security.selinux system_u:object_r:fixed_disk_device_t:s0
udevd(28299) vfs_setxattr 0x5/166267 0 security.selinux system_u:object_r:fixed_disk_device_t:s0

All in all, I’m impressed with SystemTap. It’s a simple, strongly-typed, sane programming language with type inference. Thankfully Python was not an influence on it.


Filed under Uncategorized