Tag Archives: libguestfs

Half-baked ideas: qemu -M container

For more half-baked ideas, see the ideas tag.

Containers offer a way to do limited virtualization with fewer resources. But a lot of people have belatedly realized that containers aren’t secure, and so there’s a trend for putting containers into real virtual machines.

Unfortunately qemu is not very well suited to just running a single instance of the Linux kernel, as we in the libguestfs community have long known. There are at least a couple of problems:

  1. You have to allocate a fixed amount of RAM to the VM. This is basically a guess. Do you guess too large and have memory wasted in guest kernel structures, or do you guess too small and have the VM fail at random?
  2. There’s a large amount of overhead — firmware, legacy device emulation and other nonsense — which is essentially irrelevant to the special case of running a Linux appliance in a VM.

Here’s the half-baked idea: Let’s make a qemu “container mode/machine” which is better for this one task.

Unlike other proposals in this area, I’m not suggesting that we throw away or rewrite qemu. That’s stupid, as qemu gives us lots of useful abilities.

Instead the right way to do this is to implement a special virtio-ram device where the guest kernel can start off with a very tiny amount of RAM and request more memory on demand. And an empty machine type which is just for running appliances (qemu on ARM already has this: mach-virt).

Libguestfs people and container people, all happy. What’s not to like?

1 Comment

Filed under Uncategorized

Fedora 21 is out …

… and there is a virt-builder image available. Get a Fedora 21 VM image in a few seconds:

$ virt-builder fedora-21
[   2.0] Downloading: http://libguestfs.org/download/builder/fedora-21.xz
[   2.0] Planning how to build this image
[   2.0] Uncompressing
[  14.0] Opening the new disk
[  28.0] Setting a random seed
[  28.0] Setting passwords
virt-builder: Setting random password of root to thu1hKRoXBxBigfC
[  29.0] Finishing off

Leave a comment

Filed under Uncategorized

Tip: Enable minidumps in a Windows guest

You can use virt-win-reg to enable minidumps in Windows guests. Quite easily as it happens.

First prepare a file crashcontrol.reg containing:

; NB: This assumes CurrentControlSet == ControlSet001
; See "CurrentControlSet etc." in virt-win-reg(1)

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\CrashControl]
"AutoReboot"=dword:00000000
"CrashDumpEnabled"=dword:00000003
"DumpFile"=str(2):"%SystemRoot%\MEMORY.DMP"
"LogEvent"=dword:00000001
"MinidumpDir"=str(2):"%SystemRoot%\Minidump"
"MinidumpsCount"=dword:00000032
"Overwrite"=dword:00000001

The key fields are AutoReboot, which you probably want to set to 0 to stop the guest from automatically rebooting when it gets a BSOD, and CrashDumpEnabled for which you can read the docs here.

Then import this into the guest (which must not be running):

$ virt-win-reg --merge GuestName crashcontrol.reg

Leave a comment

Filed under Uncategorized

Mapping files to disk, part 2

Part 1

Now I’ve written the second tool of virt-bmap which lets you boot a guest and observe what files it is reading from disk. (NB if you want to try this out you will need a patched libguestfs)

The second tool is an nbdkit plugin, so to use the tool you just do:

$ nbdkit -r bmaplogger file=/tmp/win7.img bmap=/tmp/win7.bmap \
  --run ' qemu-kvm -cpu host -m 2048 -hda $nbd '

and watch the output as the guest boots. Note that the bmap file must have been prepared previously by the virt-bmap tool (see part 1).

The results are interesting. Here is Windows 7 booting (edited down for brevity):

read v /dev/sda
read p /dev/sda1
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda1 /Boot/da-DK/bootmgr.exe.mui
read f /dev/sda1 /Boot/tr-TR/bootmgr.exe.mui
read f /dev/sda1 /Boot/zh-HK/bootmgr.exe.mui
read f /dev/sda1 /Boot/zh-TW/bootmgr.exe.mui
read f /dev/sda1 /bootmgr
read v /dev/sda
read p /dev/sda1
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /Boot/da-DK/bootmgr.exe.mui
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda1 /Boot/da-DK/bootmgr.exe.mui
read f /dev/sda1 /Boot/Fonts/kor_boot.ttf
read p /dev/sda1
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /Boot/da-DK/bootmgr.exe.mui
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda1 /Boot/da-DK/bootmgr.exe.mui
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /Boot/da-DK/bootmgr.exe.mui
read f /dev/sda1 /Boot/de-DE/bootmgr.exe.mui
read p /dev/sda1
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /Boot/da-DK/bootmgr.exe.mui
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda1 /Boot/da-DK/bootmgr.exe.mui
read f /dev/sda1 /Boot/BOOTSTAT.DAT
read f /dev/sda1 /bootmgr
read f /dev/sda1 /Boot/BOOTSTAT.DAT
read v /dev/sda
read p /dev/sda2
read d /dev/sda2 /
read f /dev/sda2 /Windows/System32/Msdtc/MSDTC.LOG
read d /dev/sda2 /
read f /dev/sda2 /ProgramData/Microsoft/Search/Data/Applications/Windows/MSSres00001.jrs
read d /dev/sda2 /
read d /dev/sda2 /Users
read p /dev/sda2
read d /dev/sda2 /Windows/assembly/NativeImages_v2.0.50727_64
read d /dev/sda2 /Windows
read p /dev/sda2
read d /dev/sda2 /Windows/servicing
read d /dev/sda2 /Windows
read f /dev/sda2 /Windows/System32/config/SAM.LOG1
read p /dev/sda2
read d /dev/sda2 /Windows/System32
read p /dev/sda2
read d /dev/sda2 /Windows/System32/en-US/Licenses/_Default
read d /dev/sda2 /Windows/System32
read p /dev/sda2
read d /dev/sda2 /Windows/System32
read d /dev/sda2 /Windows/System32/Tasks/Microsoft/Windows
read d /dev/sda2 /Windows/System32
read p /dev/sda2
read f /dev/sda2 /Windows/System32/CIRCoInst.dll
read d /dev/sda2 /Windows/System32
read f /dev/sda2 /Windows/System32/clb.dll
read d /dev/sda2 /Windows/System32
read f /dev/sda2 /Windows/System32/cmmon32.exe
read d /dev/sda2 /Windows/System32
read f /dev/sda2 /Windows/System32/cryptnet.dll
read d /dev/sda2 /Windows/System32
[...]
read f /dev/sda2 /Windows/System32/iscsilog.dll
read f /dev/sda2 /Windows/System32/ksetup.exe
read d /dev/sda2 /Windows/System32
read f /dev/sda2 /Windows/System32/ksproxy.ax
read f /dev/sda2 /Windows/System32/NcdProp.dll
read d /dev/sda2 /Windows/System32
read f /dev/sda2 /Windows/System32/nci.dll
read f /dev/sda2 /Windows/System32/profsvc.dll
read d /dev/sda2 /Windows/System32
read f /dev/sda2 /Windows/System32/propsys.dll
read d /dev/sda2 /Windows/System32
read p /dev/sda2
read f /dev/sda2 /Windows/System32/winload.exe
[...]

Here is a Windows server that had McAfee (a “virus scanner”) installed:

read v /dev/sda
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /bootmgr
read v /dev/sda
read f /dev/sda2 /Program Files (x86)/McAfee/Real Time/log0.txt
read v /dev/sda
read p /dev/sda1
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /Boot/nl-NL/bootmgr.exe.mui
read f /dev/sda1 /Boot/pl-PL/bootmgr.exe.mui
read f /dev/sda1 /Boot/ru-RU/bootmgr.exe.mui
read f /dev/sda1 /Boot/zh-TW/bootmgr.exe.mui
read f /dev/sda1 /bootmgr
read f /dev/sda1 /Boot/BOOTSTAT.DAT
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /Boot/Fonts/kor_boot.ttf
read f /dev/sda1 /BOOTSECT.BAK
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /BOOTSECT.BAK
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /Boot/BOOTSTAT.DAT
read f /dev/sda1 /Boot/BCD
read f /dev/sda2 /Program Files (x86)/McAfee/Real Time/log4.txt
read f /dev/sda1 /Boot/BCD
read p /dev/sda2
read f /dev/sda2 /Program Files (x86)/Common Files/microsoft shared/DAO/dao360.dll
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda2 /Program Files (x86)/Common Files/System/msadc/adcjavas.inc
read f /dev/sda2 /ProgramData/McAfee/Common Framework/Mesh/SvcMgr_WPLCLDWA170.log
read f /dev/sda2 /Program Files (x86)/McAfee/Policy Auditor Agent/auditmanager.log
read f /dev/sda2 /Program Files (x86)/Common Files/microsoft shared/DAO/dao360.dll
read f /dev/sda2 /Program Files (x86)/McAfee/Real Time/log7.txt
read f /dev/sda2 /Program Files (x86)/MSBuild/Microsoft/Windows Workflow Foundation/v3.0/Workflow.Targets
read f /dev/sda2 /Windows/ServerEnterprise.xml
read f /dev/sda2 /Windows/inf/setupapi.dev.log
read f /dev/sda2 /Program Files (x86)/McAfee/Real Time/log7.txt
read f /dev/sda2 /Program Files (x86)/Internet Explorer/en-US/jsprofilerui.dll.mui
read f /dev/sda2 /Users/tempadmin/AppData/Local/Microsoft/Internet Explorer/Recovery/High/Last Active/{7101D2F0-982F-11E0-A584-005056A7000F}.dat
read f /dev/sda2 /Program Files (x86)/McAfee/Policy Auditor Agent/Plugins/AuEngineUpdater.dll
read f /dev/sda2 /Windows/System32/clusapi.dll
read f /dev/sda2 /Windows/System32/cmcfg32.dll
read f /dev/sda2 /Windows/winsxs/Backup/amd64_microsoft-windows-com-base_31bf3856ad364e35_6.1.7600.16385_none_69e3281e403684ea_comcat.dll_8571d1d1
read f /dev/sda2 /Windows/System32/comdlg32.dll
read f /dev/sda2 /Windows/SysWOW64/comexp.msc
read f /dev/sda2 /Program Files (x86)/McAfee/Policy Auditor Agent/Schema/linux-definitions-schema.xsd
read f /dev/sda2 /ProgramData/McAfee/Common Framework/Mesh/SvcMgr_WPLCLDWA170.log
read f /dev/sda2 /Windows/SysWOW64/C_10003.NLS
read f /dev/sda2 /Windows/SysWOW64/C_10004.NLS
read f /dev/sda2 /Windows/SysWOW64/C_20005.NLS
read f /dev/sda2 /Windows/SysWOW64/C_21025.NLS
read f /dev/sda2 /Windows/CMAgent/Installer/Providers/ExecutionEngine/providers.catalog
read f /dev/sda2 /Windows/SysWOW64/dfsrHealthReport.xsl
read f /dev/sda2 /ProgramData/McAfee/Common Framework/Mesh/SvcMgr_WPLCLDWA170.log
read f /dev/sda2 /Windows/SysWOW64/C_10003.NLS
read f /dev/sda2 /Windows/SysWOW64/C_10004.NLS
read f /dev/sda2 /Windows/SysWOW64/C_20005.NLS
read f /dev/sda2 /Windows/SysWOW64/C_21025.NLS
read f /dev/sda2 /Windows/CMAgent/Installer/Providers/ExecutionEngine/providers.catalog
read f /dev/sda2 /Windows/SysWOW64/dfsrHealthReport.xsl
read f /dev/sda2 /ProgramData/McAfee/Common Framework/Mesh/SvcMgr_WPLCLDWA170.log
read f /dev/sda2 /Windows/System32/hhctrl.ocx
read f /dev/sda2 /Program Files (x86)/McAfee/Real Time/log2.txt
read f /dev/sda2 /Windows/System32/KBDA1.DLL
read f /dev/sda2 /ProgramData/McAfee/Common Framework/Mesh/SvcMgr_WPLCLDWA170.log
read f /dev/sda2 /Windows/System32/Kswdmcap.ax
read f /dev/sda2 /Windows/SysWOW64/NOISE.CHS
read f /dev/sda2 /Windows/System32/NlsData0003.dll
read f /dev/sda2 /Windows/SysWOW64/RacRules.xml
read f /dev/sda2 /Windows/System32/ROUTE.EXE
read f /dev/sda2 /Windows/SysWOW64/en-US/tapimgmt.msc
read f /dev/sda2 /Windows/SysWOW64/en-US/tpm.msc
read f /dev/sda2 /Windows/System32/TpmInit.exe
read f /dev/sda2 /Program Files (x86)/McAfee/Policy Auditor Agent/oval.db
read f /dev/sda2 /Windows/Microsoft.NET/Framework64/v4.0.30319/ngen.log
read f /dev/sda2 /Program Files (x86)/McAfee/Policy Auditor Agent/Audit.db
read f /dev/sda2 /Windows/System32/winload.exe

I wouldn’t take any of these traces very literally right now. Our method of mapping files to disk blocks is a bit shaky, especially for ntfs-3g. However I did check the major points of the McAfee trace against the raw log and block map and it seems plausible.

Leave a comment

Filed under Uncategorized

Mapping files to disk

Wouldn’t it be cool if you could watch a virtual machine booting, and at the same time see what files it is accessing on disk:

reading /dev/sda1 master boot record
reading /dev/sda1 /grub2/i386-pc/boot.img
reading /dev/sda1 /grub2/i386-pc/ext2.mod
reading /dev/sda1 /vmlinuz
...

You can already observe what disk blocks it is accessing pretty easily. There are several methods, but a quick one would be to use nbdkit’s file plugin with the -f -v flags (foreground and verbose). The problem is how to map disk blocks to the files and other interesting objects that exist in the disk image.

How do you map between files and disk blocks? For simple filesystems like ext4 you can use the FIBMAP ioctl, and perhaps adjust the answer by adding the offset of the start of the partition. However as you get further into the boot process you’ll probably encounter complexities like LVM. There may not even be a 1-1 mapping since RAID means that multiple blocks can store a single file block, and tail packing and deduplication mean that a block can belong to multiple files. And of course there are things other than plain files: directories, swap partitions, master boot records, and boot loaders, that live in and between filesystems.

To solve this I have written a tool called virt-bmap. It takes a disk image and outputs a block map. To do this it uses libguestfs (patched) to control an nbdkit instance, reading each file and recording what blocks in the disk image are accessed. (It sounds complicated, but virt-bmap wraps it up in a simple command line tool.) The beauty of this is that the kernel takes care of the mapping for us, and it works no matter how many layers of filesystem/LVM/RAID are between the file and the underlying device. This doesn’t quite solve the “RAID problem” since the RAID layers in Linux are free to only read a single copy of the file, but is generally accurate for everything else.

$ virt-bmap fedora-20.img
virt-bmap: examining /dev/sda1 ...
virt-bmap: examining /dev/sda2 ...
virt-bmap: examining /dev/sda3 ...
virt-bmap: examining filesystem on /dev/sda1 (ext4) ...
virt-bmap: examining filesystem on /dev/sda3 (ext4) ...
virt-bmap: writing /home/rjones/d/virt-bmap/bmap
virt-bmap: successfully examined 3 partitions, 0 logical volumes,
           2 filesystems, 3346 directories, 20585 files
virt-bmap: output written to /home/rjones/d/virt-bmap/bmap

The output bmap file is a straightforward map from disk byte offset to file / files / object occupying that space:

1 541000 541400 d /dev/sda1 /
1 541400 544400 d /dev/sda1 /lost+found
1 941000 941400 f /dev/sda1 /.vmlinuz-3.11.10-301.fc20.x86_64.hmac
1 941400 961800 f /dev/sda1 /config-3.11.10-301.fc20.x86_64
1 961800 995400 f /dev/sda1 /initrd-plymouth.img
1 b00400 ef1c00 f /dev/sda1 /grub2/themes/system/background.png
1 f00400 12f1c00 f /dev/sda1 /grub2/themes/system/fireworks.png
1 1300400 1590400 f /dev/sda1 /System.map-3.11.10-301.fc20.x86_64

[The 1 that appears in the first column means “first disk”. Unfortunately virt-bmap can only map single disk virtual machines at present.]

The second part of this, which I’m still writing, will be another nbdkit plugin which takes these maps and produces a nice log of accesses as the machine boots.

3 Comments

Filed under Uncategorized

Finding bugs in hivex with afl-fuzzer

Michał Zalewski’s blog has been even more interesting than usual lately: first he discovered that running “strings” on untrusted files can be exploitable, then he wrote an interesting article about pulling JPEG files out of thin air. In both cases he used his very practical fuzzer, American fuzzy lop (abbreviated to “afl”, also a breed of rabbit in case you were wondering).

It’s a very practical, easy to use, and dangerously good fuzzer. I’ve been running it on hivex — my library for reading the Windows registry, and found 3 crasher bugs within 48 hours (one of them within minutes) [Update: This turned out to be user error because I was mixing a newly built binary with the installed libhivex.so library. However it still demonstrated its effectiveness at finding bugs.]

Here’s how you too can exploit hivex and many other programs:

  1. Install afl (Fedora package review).
  2. Configure and build hivex like this:
    CC=/usr/bin/afl-gcc ./configure
    make
    
  3. Copy the minimal hive to a new directory:
    mkdir input
    cp lib/minimal input/
    
  4. Run afl-fuzz:
    libtool --mode=execute afl-fuzz -i input -o output -f testme ./xml/hivexml testme
    

Sit back and watch afl find inputs that crash your program (see the output/crashes directory that afl creates).

Now my day will be spent examining the hivex bugs and submitting patches and/or CVEs for them.

2 Comments

Filed under Uncategorized

Tip: Read guest disks from VMware vCenter using libguestfs

virt-v2v can import guests directly from vCenter. It uses all sorts of tricks to make this fast and efficient, but the basic technique uses plain https range requests.

Making it all work was not so easy and involved a lot of experimentation and bug fixing, and I don’t think it has been documented up to now. So this post describes how we do it. As usual the code is the ultimate repository of our knowledge so you may want to consult that after reading this introduction.

Note this is read-only access. Write access is possible, but you’ll have to use ssh instead.

VMware ESXi hypervisor has a web server but doesn’t support range requests, so although you can download an entire disk image in one go from the ESXi hypervisor, to random-access the image using libguestfs you will need VMware vCenter. You should check that virsh dumpxml works against your vCenter instance by following these instructions. If that doesn’t work, it’s unlikely the rest of the instructions will work.

You will need to know:

  1. The hostname or IP address of your vCenter server,
  2. the username and password for vCenter,
  3. the name of your datacenter (probably Datacenter),
  4. the name of the datastore containing your guest (could be datastore1),
  5. .. and of course the name of your guest.

Tricky step 1 is to construct the vCenter https URL of your guest.

This looks like:

https://root:password@vcenter/folder/guest/guest-flat.vmdk?dcPath=Datacenter&dsName=datastore1

where:

root:password
username and password
vcenter
vCenter hostname or IP address
guest
guest name (repeated twice)
Datacenter
datacenter name
datastore1
datastore

Once you’ve got a URL that looks right, try to fetch the headers using curl. This step is important! not just because it checks the URL is good, but because it allows us to get a cookie which is required else vCenter will break under the load when we start to access it for real.

$ curl --insecure -I https://....
HTTP/1.1 200 OK
Date: Wed, 5 Nov 2014 19:38:32 GMT
Set-Cookie: vmware_soap_session="52a3a513-7fba-ef0e-5b36-c18d88d71b14"; Path=/; HttpOnly; Secure; 
Accept-Ranges: bytes
Connection: Keep-Alive
Content-Type: application/octet-stream
Content-Length: 8589934592

The cookie is the vmware_soap_session=... part including the quotes.

Now let’s make a qcow2 overlay which encodes our https URL and the cookie as the backing file. This requires a reasonably recent qemu, probably 2.1 or above.

$ qemu-img create -f qcow2 /tmp/overlay.qcow2 \
    -b 'json: { "file.driver":"https",
                "file.url":"https://..",
                "file.cookie":"vmware_soap_session=\"...\"",
                "file.sslverify":"off",
                "file.timeout":1000 }'

You don’t need to include the password in the URL here, since the cookie acts as your authentication. You might also want to play with the "file.readahead" parameter. We found it makes a big difference to throughput.

Now you can open the overlay file in guestfish as usual:

$ export LIBGUESTFS_BACKEND=direct
$ guestfish
><fs> add /tmp/overlay.qcow2 copyonread:true
><fs> run
><fs> list-filesystems
/dev/sda1: ext4
><fs> mount /dev/sda1 /

and so on.

4 Comments

Filed under Uncategorized