Tag Archives: virtualization

It’s all about memory

The thing that stops me from running lots and lots of virtual machines is the amount of RAM I can fit into a server.

My current build server runs 9 VMs taking just over 7 GB in total. The host has 8 GB of RAM.

The 4 cores on the host are virtually idle. Certainly CPU-wise there is nothing stopping us running 16 or possibly even 32 VMs.

RAM is the problem. The server can take 16 GB maximum, at a cost (for cheap desktop RAM) of only about $400, and I get the original 8 GB back to recycle somewhere else. Unfortunately the larger RAM would be slower. For most consumer hardware 8 or 16 GB seems to be the limit anyway.

Ulrich Drepper has a very good analysis of DDR RAM, why larger sizes are necessarily slower, and why you can’t design single socket systems that take arbitrarily huge amounts of RAM.

I’m left doing workarounds: reduce the amount of RAM assigned to each guest, keep guests paused or powered off when not directly in use, move guests between servers, … Not ideal.


Filed under Uncategorized

Tip: Debugging the early boot process with qemu and gdb

Update: A much easier way is to use gdbserver.

Start qemu with the following parameters:

$ qemu-system-x86_64 -s -S -m 512 -hda winxp.img

And connect with gdb like this:

$ gdb
(gdb) target remote localhost:1234
(gdb) set architecture i8086
(gdb) break *0x7c00
(gdb) cont

This will breakpoint at 0x7c00, which is when the boot sector has been loaded into memory by the BIOS and control is passed to the boot sector.

You can use ordinary gdb commands to disassemble and debug the guest.


Filed under Uncategorized

A 685K Fedora appliance

Using supermin appliances we can make some very small to download Fedora appliances. These ones are under 700K (yes, that’s “K” not “M”).

For Fedora 15, use this link:

For Fedora 16, use this link:

For Rawhide, use this link:

You will need ~600 MB free space in /var/lib since that is where the real appliance gets built. Just install the RPM and run sudo boot-a-fedora-appliance. Then read that script and the README file.

Upstream source:

Leave a comment

Filed under Uncategorized

RHEL virtualization getting started guide

The new Red Hat Enterprise Linux Virtualization Getting Started Guide, which I worked on, is essential reading if you want to find out how to start out using KVM on RHEL 6.


Filed under Uncategorized

Debian/Ubuntu package for Oz (an excellent VM builder)

Oz is an excellent virtual machine builder. I reviewed it here.

This is a temporary Debian (and possibly Ubuntu) package for Oz:


I’ll put up a more permanent site for Debian packages later when we’ve decided where to host them.

Note there is no separate source archive of this right now. The source is upstream git + the Debian patch that I posted on the mailing list (but the mailing list archives seem to be broken — fixing it). You can build the exact same package yourself by checking out git, applying the Debian patch, and doing:

debuild -i -uc -us -b


Filed under Uncategorized

Fedora 16 Virtualization test day

Come and help us test the virtualization features in Fedora 16, this Thursday (15th September).

Go here for more details:



Filed under Uncategorized

Playing with oz-install

oz-install (see also: Oz) is a command line program for building operating system instances. It can automatically install a fairly wide variety of OSes, including Windows.

Using it is pretty effortless. You first need to write a small template file, which just describes the name and version of the OS you want to install and where to get the ISO from. Here is one for Fedora 14:

    <install type='iso'>
  <description>Fedora 14 x86_64 template</description>

By default oz-install will write the destination VM to /var/lib/libvirt/images, so you either need to check you’ve got enough free disk space under there, or else change the output path in /etc/oz/oz.cfg.

Using oz-install is simple, just do:

# oz-install fedora.xml
Libvirt XML was written to fedora14_x86_64Aug_30_2011-13:53:07

(oz-install requires root, but there is a feature request to remove this limitation)

oz-install writes a disk image and a libvirt XML file that you can use to immediately boot the guest:

# virsh define fedora14_x86_64Aug_30_2011-13:53:07
Domain fedora14_x86_64 defined from fedora14_x86_64Aug_30_2011-13:53:07

# virsh start fedora14_x86_64
Domain fedora14_x86_64 started

# virt-viewer fedora14_x86_64

I don’t know what root password is given to new Oz guests, but I was easily able to edit it out using guestfish.

Note that serious Oz users will want to further customize the guest using oz-customize, rather than booting it like I did. The workflow is that you use oz-install once to create the operating system template. Then you can duplicate the template, customize it (oz-customize) and deploy it. The copy-and-customize step is much quicker than installing a whole new guest from scratch each time.

Update: Chris tells me that the default root password is ozrootpw. You can also set it in the template description file by adding a <rootpw> element.

1 Comment

Filed under Uncategorized

Fun new virt tools: virt-dmesg and virt-uname

About 4 years ago I wrote some code that peeks into the memory of guests, reads out the kernel, and finds out useful stuff: symbols, process tables, kernel messages, network interfaces and so on. Actually I should say at this point you can already do this using the excellent crash tool, and if you have the debuginfo for all your guest kernels, you should use that tool. My code was trying to be more ambitious, because it heuristically examined the kernel memory, guessing the location of various structures.

Well, that code was not so successful. Heuristics only take you so far when you’re looking at something as complex and variable as the process table. One day I hope to make a big breakthrough to make it all possible.

Nevertheless, some things worked quite well: it was actually pretty easy to snoop out the kernel symbols, and once you’ve got those some of the more static kernel structures are easily accessible.

For the past few days I’ve been working on two tools derived from that.

virt-dmesg shows you the kernel messages from a running Linux virtual machine:

# virt-dmesg F14x64 | tail -5
<6>[   11.609206] lo: Disabled Privacy Extensions
<6>[   11.634902] ip6_tables: (C) 2000-2006 Netfilter Core Team
<6>[   15.286998] eth0: link up, 100Mbps, full-duplex, lpa 0x05E1
<7>[   21.898345] mtrr: no MTRR for f0000000,100000 found
<7>[   25.714129] eth0: no IPv6 routers present

virt-uname shows the utsname structure:

# virt-uname Fedora14
Linux f14x64.home.annexia.org #1 SMP
Mon Feb 7 07:06:44 UTC 2011 x86_64 (none)

The main page for virt-dmesg and virt-uname is here, and there is also a git repository.

You will need a bunch of patches against libvirt to make it work. I have updated libvirt in Rawhide with all the required patches, and without that version of libvirt it definitely won’t work at all.

It works for many, but not all, guests that I’ve tried. The main problem is with guests that don’t enable the full kallsyms feature. If the guest doesn’t have all of the kernel configuration options below enabled, then you may have to fall back to dumping out the kernel and grepping through it with “strings”.


Leave a comment

Filed under Uncategorized

What is sVirt?

There seems to be a lot of confusion about the “sVirt” feature that we added in Fedora 11. Really it’s very simple, and in this posting I hope to explain it in very simple terms.

Let’s start with the problem that sVirt tries to solve: If you run lots of KVM virtual machines on a single host, then probably all those qemu processes are running as the same user. They could all be running as root (very bad!). Better, they might all be running as a separate qemu.qemu user/group. Also, any disks they are using are probably chowned to qemu.qemu too.

The problem is that the interface between the virtual machines and the containing qemu process is very complicated and hacked together in C. It’s very likely that this boundary is full of undiscovered insecurities that allow a user in the virtual machine to take over the qemu process — in other words, to escape from the confinement provided by the hypervisor. (Xen and other hypervisors have similar problems, this is not something that’s special to KVM).

If all your qemu processes are running as the same user, there is literally no protection between the virtual machines if one is compromised like this. Two processes running as the same user can send signals, insert data into each other using ptrace, and lots more — if one qemu process “goes bad”, you’ve lost control of all the other qemu processes on the host. Furthermore, because all the disk images (and other resources) were accessible by the single qemu.qemu user, you’ve also lost all those too.

What’s the solution? Well, sVirt of course. One thing you could do is to run all the qemu processes as different users, but that’s not very convenient because it would mean reserving a block of hundreds of UIDs and GIDs. Step forward SELinux: without needing to reserve anything, you can give each qemu process a different SELinux label. This firstly prevents a compromised qemu from attacking other processes, and also allows you to label the precise set of resources that each process can see — so a compromised qemu can only attack its own disk images.

You can see an example of sVirt labels on this page.

What happens if you turn off SELinux (or use a distribution that doesn’t have libvirt, SELinux or sVirt in the first place)? You’re trusting that huge, hacked together boundary between the VM and the hypervisor to keep you safe. Good luck.


Filed under Uncategorized

Tip: Change the background image in a Windows VM

Thanks to Tom Horsley who worked out how to do this for Windows XP guests (the technique is probably different for other versions of Windows).

Here is Tom’s script and here are more of his KVM tips.


Filed under Uncategorized