Tag Archives: bugs

Fuzz-testing libguestfs inspection code

There are a lot of security issues with dealing with untrusted disk images especially since for historical reasons a lot of the code used to parse filesystems sits in the kernel. Libguestfs avoids these by wrapping the kernel code inside a VM (and that VM inside an sVirt container if you’re using Fedora or RHEL).

However the library side of things could still be vulnerable, especially complicated operations like inspection. Last week we found several vulnerabilities in inspection which could allow an untrusted guest to perform a denial of service attack on a host.

The first vulnerability was identified by Coverity. The second was found by Olaf Hering by looking at similar code paths.

This made me wonder if we could find more inspection bugs semi-automatically. To do this I’ve written an inspection fuzz tester.

The idea is we run inspection on an empty disk image. Normally this wouldn’t find any operating systems. But we intercept certain libguestfs calls (which happen as a side-effect of inspection) and use them to create fake operating system files on the fly.

To give you an example: Inspection might look for a file called /etc/redhat-release and then try to parse it. To do this it will first test if the file exists (guestfs_is_file ("/etc/redhat-release")) and if it does read it. In the empty disk this file won’t exist, but we capture the is_file call, randomly create a file, and then see what happens when inspection tries to parse it.

Libguestfs has a trace mechanism but if we decided to do this sort of thing regularly we’d probably want to add a cleaner way to find the arguments and perhaps even replace the return value from a method call.

The result is a fuzz tester which now runs as part of the ordinary test suite.

I also ran many tens of thousands of iterations over the weekend. The test found Olaf Hering’s bug, which is encouraging, but it didn’t find any other bugs, which means there is room for refinement of the test. In particular I think we could push more malformed registry hives at the inspection code to see what it does.

Leave a comment

Filed under Uncategorized

libguestfs build — an open ended problem

librarian made a very true observation (Google translate) about libguestfs. It’s a Swiss army chainsaw, but it’s damn hard to build from source.

With RHEL and Fedora I’ve made it my aim that no one should need to build libguestfs from source, because we offer the highest quality packages with every feature compiled in. I also build Debian and Ubuntu packages when I can and until someone steps up to do that.

But why is libguestfs a difficult package to build?

The primary reason is that we package up, make an API for, and rigorously test, something like 200 different Linux packages. Essentially if you use (say) the guestfs_part_* API then in the background you’re using parted. If you’re using another API, you might be using e2fsck or resize2fs or lvm or grep or file or the kernel or any one of dozens of other programs. And to compound the problem, we don’t just “ship and forget”. We test these programs, and if they break, then we break. Our test suite has about 600 different tests and takes 2 hours to run.

And we test against Fedora Rawhide. The latest and buggiest.

Consequently we hit all the new bugs. Just today I hit a Linux 3.0 bug and another kernel/ftrace bug. Two weeks ago it was a bug in the file command, another bug in udev on Debian, and you can never exclude the possibility of stupidity by Ubuntu kernel maintainers.

It’s routine that I discover qemu, kernel and other bugs for the first time, because often a libguestfs build in Koji is the first build that boots up and runs the new software.

So what’s my point? It would be good if the Fedora kernel and qemu maintainers didn’t just push out a new package, but they tested that one can run inside the other. But while that would improve the situation for me, the real problem is that integrating software is hard, and it’s unfortunate that libguestfs has got into a situation where we are the first people to integrate and run Rawhide.

Leave a comment

Filed under Uncategorized

What can affect a process?

I’ve been following an interesting thread on fedora-devel which set me thinking. What is the complete list of different things that can affect the way your process runs?

Here’s my list below. If you have any other ideas, post a comment and I’ll keep this list updated.

  1. Environment variables. Obviously there are direct effects, like if $PATH is different then you may end up running different sub-processes. But there are more subtle differences like what happens if the environment is too large or completely empty? Also $LD_* variables can make a big difference to what is inside your process.
  2. ulimits. Too small and your program could fail to allocate memory or fail to open a file.
  3. Signal masks. Often overlooked, but I’ve hit this one: If a signal is masked, your program can behave quite differently. There is a famous bug where SIGPIPE was masked in the whole of Fedora, because some early program (login) was using dbus which promiscuously masked the signal, then login was forking every other program with this signal masked.
  4. Program arguments. You could put this in the “too obvious” class if you want, but consider also argv[0] which might affect your program but not be an immediately visible change.
  5. The PID. I have actually seen this: a program (sshd) was trying to create some lock file, something like /var/lock/sshd.<pid> at boot time in order to ensure only one instance was running. It was consistently failing to start sshd at boot. It took me some time to work out that because the boot was exactly predictable (and thus the PID was always the same), it was falling over its own lock file left from last time the machine was shut down.
  6. The file descriptors. Does the program change behaviour if fds 0, 1, 2 (stdin, stdout, stderr) are not open? How about if other open fds are leaked from the parent process?
  7. Current working directory. Affects what files are opened by relative paths. I guess you could include the chroot here too, but that is quite an obvious change.
  8. Number of other processes. This is like an “unofficial” ulimit, since as normally configured Linux will only allow 32766(?) processes (less PID 0 which is reserved and PID 1 for init).
  9. UIDs and GIDs. If these are very large, Bad Things can happen. External utilities like cpio and tar will fail.
  10. SELinux context. (Suggested by David Malcolm) One thing to note is that the SELinux context of a root login can be different from the context of, say, a daemon started at boot.
  11. Wallclock time or other timers. (Alexander E. Patrakov)
  12. Filesystem journalling mode, filesystem type. (see Bruno Wolff’s comment)

That’s all I can think of for now. Post a comment if you can think of any more.


Filed under Uncategorized

Tip: daily Bugzilla reports

This is my inbox, and it sucks:

Bugzilla is like a black hole for bugs. The search tools fail so badly it’s often impossible to find a bug that you were working on the day before. It’s slow, clumsy, and disorganized.

But one bright point is it has a reasonable command line reporting tool also available as a Fedora package. So I decided yesterday to write a Bugzilla report script and have it email me daily from cron.

The starting point is to identify “bugs of interest”. The bugs of interest to me are:

  • bugs I reported
  • bugs I am assigned to fix
  • bugs I’m CC’d on

I put some thought into this set of criteria for bugs:

  1. I should be able to register an interest in any bug: Yes, by adding myself to the CC field.
  2. I should be able to unregister an interest: Yes, by removing myself from the CC, or closing or reassigning those bugs where I’m reporter or assignee.
  3. It shouldn’t tell me about bugs I’m not interested in: Yes, I would be reporter, assignee or in the CC list for any bug related to me or a package I maintain.
  4. It shouldn’t miss out any bugs I might be interested in: Yes, if I’ve ever added a comment, I will be in the CC field at least, and I can always unregister myself from any I no longer care about.

With the command line tool, getting the raw list of bug IDs is very simple.

for flag in -r -c -a; do
  bugzilla query $flag $email -i
done | sort -n -u

(replacing $email with your email address, or even mine if you so care).

That pulls out 781 unique bug IDs, the oldest being one I don’t remember reporting and the most recent being a virt-v2v bug. (Note this includes CLOSED bugs which in the script are ignored).

Now I take the list of bug IDs and pull out the other fields I want from the Bugzilla database. The command below is just an illustrative example of what the script does:

$ bugzilla query -b "86308,601092" \
  --outputformat="%{bug_id}|%{bug_status}|%{product}|%{component}|%{short_desc}" |
  sort -n -t"|"
86308|CLOSED|Red Hat Web Site|Other|Red Hat Command Centre site is down
601092|NEW|Red Hat Enterprise Linux 5|libguestfs|[RFE]Incorrect error msg popped up when missing "-f" in v2v command

As far as I know you have to guess what the %-fields are called. I also had to choose a separator character which wouldn’t appear in any field except the short_desc (which is always the last field), since fields like the product name can and do contain spaces.

The rest of the script is “merely” formatting this whole lot into a nice looking email report:

Bugs in OPEN states:

*** coccinelle ***

  In Fedora:
    502185 coccinelle segfaults on ppc64
    579457 coccinelle-0.2.3rc1 is available

*** e2fsprogs ***

  In Red Hat Enterprise Linux 5:
    518190 mke2fs -t option in RHEL5 differs vs upstream, leading to confusion


You can download the script here: http://oirase.annexia.org/tmp/bugs-report.ml.txt


Filed under Uncategorized