Using American Fuzzy Lop on network clients

Previously I’ve fuzzed hivex and nbdkit using my favourite fuzzing tool, Michał Zalewski’s American Fuzzy Lop (AFL).

AFL works by creating test cases which are files on disk, and then feeding those to programs which have been specially compiled so that AFL can trace into them and find out which parts of the code are run by the test case. It then adjusts the test cases and repeats, aiming to run more parts of the code and find ways to crash the program.

This works well for programs that parse files (like hivex, but also binary parsers of all sorts and XML parsers and similar). It can also be used to fuzz some servers where you can feed a file to the server and discard anything the server sends back. In nbdkit we can use the nbdkit -s option to do exactly this, making it easy to fuzz.

However it’s not obvious how you could use this to fuzz network clients. As readers will know we’ve been writing a new NBD client library called libnbd. But can we fuzz this? And find bugs? As it happens yes, and ooops — yes — AFL found a remote code execution bug allowing complete takeover of the client by a malicious server.

The trick to fuzzing a network client is to do the server thing in reverse. We set up a phony server which feeds the test case back to the client socket, while discarding anything that the client writes:

This is wrapped up into a single wrapper program which takes the test case on the command line and forks itself to make the client and server sides connected by a socket. This allows easy integration into an AFL workflow.

We found our Very Serious Bug within 3 days of fuzzing.

Tip: Enable minidumps in a Windows guest

You can use virt-win-reg to enable minidumps in Windows guests. Quite easily as it happens.

First prepare a file crashcontrol.reg containing:

; NB: This assumes CurrentControlSet == ControlSet001
; See "CurrentControlSet etc." in virt-win-reg(1)

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\CrashControl]
"AutoReboot"=dword:00000000
"CrashDumpEnabled"=dword:00000003
"DumpFile"=str(2):"%SystemRoot%\MEMORY.DMP"
"LogEvent"=dword:00000001
"MinidumpDir"=str(2):"%SystemRoot%\Minidump"
"MinidumpsCount"=dword:00000032
"Overwrite"=dword:00000001

The key fields are AutoReboot, which you probably want to set to 0 to stop the guest from automatically rebooting when it gets a BSOD, and CrashDumpEnabled for which you can read the docs here.

Then import this into the guest (which must not be running):

$ virt-win-reg --merge GuestName crashcontrol.reg

Finding bugs in hivex with afl-fuzzer

Michał Zalewski’s blog has been even more interesting than usual lately: first he discovered that running “strings” on untrusted files can be exploitable, then he wrote an interesting article about pulling JPEG files out of thin air. In both cases he used his very practical fuzzer, American fuzzy lop (abbreviated to “afl”, also a breed of rabbit in case you were wondering).

It’s a very practical, easy to use, and dangerously good fuzzer. I’ve been running it on hivex — my library for reading the Windows registry, and found 3 crasher bugs within 48 hours (one of them within minutes) [Update: This turned out to be user error because I was mixing a newly built binary with the installed libhivex.so library. However it still demonstrated its effectiveness at finding bugs.]

Here’s how you too can exploit hivex and many other programs:

1. Install afl (Fedora package review).
2. Configure and build hivex like this:

   CC=/usr/bin/afl-gcc ./configure
   make
   

3. Copy the minimal hive to a new directory:

   mkdir input
   cp lib/minimal input/
   

4. Run afl-fuzz:

   libtool --mode=execute afl-fuzz -i input -o output -f testme ./xml/hivexml testme
   

Sit back and watch afl find inputs that crash your program (see the output/crashes directory that afl creates).

Now my day will be spent examining the hivex bugs and submitting patches and/or CVEs for them.

hivexfs

If you have ever felt the need to mount a Windows registry hive as a FUSE filesystem, Sergey Trubin’s hivexfs may be the project for you!

libguestfs 1.12.6 for Debian

Thanks to the tireless work of Hilko Bengen, libguestfs 1.12.6 is now available as an official Debian package.

Also, you can compile hivex on Mac OS X and Windows, thanks to Alex Nelson and Gillen Daniel respectively.

Ruby bindings for Hivex

Hivex is a library for reading and writing Windows Registry “hive” files. New in version 1.3.0 and Fedora 16 is the ability to access the library from Ruby.

As an example, first grab some hive files from a Windows virtual machine. The simplest way is using virt-copy-out:

# virt-copy-out -a win.img \
    'win:c:\windows\system32\config' .
# ls config/
...
SOFTWARE
SYSTEM
...

Using the following Ruby script you can extract and display registry keys from the hive files:

#!/usr/bin/ruby

require 'hivex'

h = Hivex::open("config/SOFTWARE", {})

# Use this instead if you want to make changes:
# h = Hivex::open("config/SOFTWARE", { :write => 1 })

root = h.root()
node = h.node_get_child(root, "Microsoft")
if node.nil? then
  puts "no HKLM\\SOFTWARE\\Microsoft node: Probably not the correct hive"
end

node = h.node_get_child(node, "Windows NT")
node = h.node_get_child(node, "CurrentVersion")
val = h.node_get_value(node, "ProductName")

hash = h.value_value(val)
puts "Windows product name:", hash[:value]

Today, in other projects …

virt-top 1.0.6 – utility for displaying virtualization stats, like ‘top’

hivex 1.2.8 – library and tools for reading and writing Windows Registry hive files

Tip: Change the background image in a Windows VM

Thanks to Tom Horsley who worked out how to do this for Windows XP guests (the technique is probably different for other versions of Windows).

Here is Tom’s script and here are more of his KVM tips.

hivex 1.2.5 released

The latest version of hivex — the library for extracting and modifying Windows Registry hive files has been released. You can get the source from here.

I spent a lot of time examining real hive files from Windows machines and running the library under the awesome valgrind tool, and found one or two places where a corrupt hive file could cause hivex to read uninitialized memory. It’s not clear to me if these are security issues — I think they are not — but everyone is advised to upgrade to this version anyway.

hivex would be a great candidate for fuzz testing if anyone wants to try that.

Tip: Code for getting DHCP address from a virtual machine disk image

Previously (1) and previously (2) I showed there are many different ways to get the IP address from a virtual machine.

The example below shows one way to use libguestfs and hivex from a C program (virt-dhcp-address) to get the DHCP address that a virtual machine has picked up.