Tip: Configure guest with filesystem directly on a host device

Xen lets you assign a host block device as a guest partition, synthesizing the partition table for you. So for example, the host /dev/VG/LV might appear in the guest as /dev/xvda1 with a filesystem directly on it. This means from the host you just see a filesystem, which you can create directly, mount etc. Some people like this and others like me have guests already configured like this.

It is possible to configure a libvirt/KVM host this way, and this post describes how. It is somewhat more manual than using virt-install. In fact I will assume that you either already have guests configured this way, or you know how to prepare a guest filesystem (eg. using debootstrap) directly onto a device like this.

There are two points about such a configuration on KVM.

Number one: the kernel and initrd of the guest live “outside” the guest, on the host. This can be beneficial, eg. if you want all your guests to have a common kernel which you will manage, compile and upgrade centrally. Number two is that KVM cannot synthesize the partition table like Xen, so inside the guest it’s going to see a filesystem on /dev/vda directly, with no partition table. Linux will work just fine. Windows wouldn’t, but that doesn’t matter because Windows cannot be configured like this anyway.

Decide where to put your external kernel and initrd on the host, and then configure libvirt like this:

<domain type='kvm'>
  <name>guest</name>
  ...
  <os>
    <type arch='x86_64' machine='pc'>hvm</type>
    <kernel>/usr/local/etc/virt/vmlinuz</kernel>
    <initrd>/usr/local/etc/virt/initrd</initrd>
    <cmdline>ro root=/dev/vda</cmdline>
    <boot dev='hd'/>
  </os>
  ...
  <devices>
    <disk type='block' device='disk'>
      <source dev='/dev/VG/LV'/>
      <target dev='vda' bus='virtio'/>
    </disk>

The kernel, initrd, cmdline elements are what libvirt calls “direct kernel boot”.

Secondly of course you need your guest filesystem directly on the device or logical volume. Its fstab should be set up accordingly, plus any other configuration files it needs. (guestfish can of course safely create, view and edit guest filesystems which are configured in this manner).

In fact I use several production guests configured like this, for historical reasons (they came from a Xen server). However I find in general this a more clumsy way to organize guests. It might scale better at very high-end configurations, if you wrote a lot of custom tools, but the vast majority of users don’t have systems on such a scale.

There is also a security issue: although you can mount a guest filesystem directly on the host this way, it’s likely that you shouldn’t. Only this week I found a kernel-crasher, possibly exploitable, in the minix filesystem driver. Use libguestfs to put a safe barrier between your guests and your host.

Prebuilt distributions part 3

[This is the final article in a three part series. Read part 1 and part 2.]

In this part I was going to discuss the different installation options, and I’ll give a quick introduction to each at the end. But you know? What I really need is this tool:

+--------------------------------------------------------+
| Rich's Amazing Bug Environment Reproducer Tool!        |
+--------------------------------------------------------+
| Bugzilla number: # [545116] [ Go ]                     |
|                                                        |
| Preselecting Fedora Rawhide image.  If this is wrong,  |
| choose the right image from the list below:            |
|                                                        |
| [X] Fedora Rawhide (x86-64)                            |
| [ ] Fedora 12                                          |
| [ ] Fedora 11                                          |
| [ ] Ubuntu 9.11                                        |
| [ ] Ubuntu 9.04                                        |
| [ ] Debian 6.0                                         |
| [ ] Debian 5.0    [Download more distributions]        |
|                                                        |
| [X] x86-64   [ ] i386                                  |
|                                                        |
| [Create  virtual machine now]                          |
+--------------------------------------------------------+

The Create virtual machine now button must take no more than 60 seconds to deliver the virtual machine, plus maybe another 60 seconds to boot it and give me the login prompt. I can then go in, configure it and reproduce the bug, and snapshot it, so I can go back and test fixes later. (I’d like the snapshot workflow to go through the user interface too, and ponies).

The only approach that doesn’t take ages to install is to have the distributions prebuilt as small disk images locally (as described already in part 2).

Note stuff I don’t want to have to specify: storage, download URLs, VM names, etc etc.

Why doesn’t this exist already, or perhaps it does?

I’d also like to suggest that the resizable disk image should become yet another distribution format that Linux distros provide. Many do already, albeit buried inside a live CD.

---

As promised, my quick summary of installation methods:

virt-install

virt-install is a great command line tool for installing new VMs. I have these two commands pretty much permanently in my command history:

# lvcreate -n F13Rawhidex64 -L 10G vg_trick
# virt-install -v -n F13Rawhidex64  --accelerate \
    -r 512 -f /dev/vg_trick/F13Rawhidex64 \
    --cdrom Fedora-12-x86_64-DVD.iso

The commands above lead to a manual installation, but you can also specify a kickstart file, although the syntax for doing this is not obvious and not documented in the manual page.

Nevertheless, even an install from a CD-ROM on the local drive (as in the example above) is a slow process, taking anything up to an hour.

kickstart

Kickstart is the method used to automate Anaconda installs, ie. installs of all Fedora, Red Hat and derived distributions. It’s a little unwieldy because you have to write a separate kickstart file, then host that on a web server (so the VM can find it), then add a special magic kernel parameter when booting the VM: ks=http://local.example.com/kickstart.ks

I found in the past that it’s better to keep a collection of kickstart files around and iterate them slowly so you can find out what works and what doesn’t.

Also, kickstarts are automatic, but just as slow as manual installation.

debootstrap

Debian’s debootstrap (and the Fedora equivalent I wrote called febootstrap) is, as far as I know, the standard way to install a new Debian instance from an existing Debian instance. Unlike our VM installs above, debootstrap creates a “chroot” directory, and what you do with that directory is then up to you. Usually you would debootstrap into some mounted device. debootstrap is very simple to use and driven completely from the command line.

It’s not particularly fast. Because it downloads, unpacks and installs each package, it can take an hour or more to run.

ubuntu-vm-builder

This is Ubuntu’s wrapper around debootstrap and libvirt/qemu.

cobbler

Cobbler is a network install server which lets you pxeboot and configure a network of machines (and virtual machines) from a central server. It has lots of wonderful features — go and read the web site.

Prebuilt distributions part 2

In part 1 I discussed how these days Linux Live CDs usually come with a prebuilt disk image of the distro which is simply copied over to the hard disk during installation. (The “old” method was to rpm/dpkg-install the packages which is much more time-consuming). However my first test wasn’t very successful because I was using the “cp” command to copy files.

Anaconda (the Fedora installer) is smarter than this. It “dd”s the prebuilt disk image to the hard disk and then uses an ext2/3/4 utility called resize2fs to expand it to the correct size.

I changed the previous guestfish script to take this approach.

The new/Anaconda approach is much faster. Our total time is down from over 18 minutes to 2½ minutes (approximately 2 minutes for the “dd”, 2 seconds for the resize2fs, and the rest of the time taken doing the partitioning and LVM creation).

Unfortunately we have to leave Ubuntu behind at this point. Ubuntu ships with a squashfs, and I’m not aware of any way to turn this into an ext3 partition efficiently (except to use “cp” which we showed in part 1 was very slow). The new script only works with Fedora Live CD ISOs.

The new script is after the cut.

Prebuilt distributions part 1

Previously I took a look at unpacking Fedora and Ubuntu live CDs to find out what’s inside them and to ask the question can we use the prebuilt filesystem image that these live CDs contain to quickly create a Fedora or Ubuntu “all-defaults” virtual machine?

This is my first attempt, and it’s not successful, but it does demonstrate a large and interesting guestfish script doing a non-trivial amount of work.

This script:

1. mounts the prebuilt filesystem from either a Fedora or Ubuntu live CD
2. creates a disk image with a 200 MB /boot partition and a single / (root) logical volume covering the remainder of the disk
3. uses the cp -a command to recursively copy the prebuilt filesystem to the disk

Where it fails is that “cp” isn’t very fast. On my local machine it took 18 minutes to copy all the files across, which means this isn’t a practical “instant install” method. (I didn’t in the end try to boot the final disk image).

In part 2 this week, I’ll look at the approach that anaconda takes: It dd’s the disk image and then runs resize2fs on it to expand it into the available space.

In part 3 I’ll compare this approach to others: virt-install, manual installation, kickstart, cobbler, debootstrap and ubuntu-vm-builder.

The script itself follows after the cut:

febootstrap: Fedora equivalent of debootstrap

As I mentioned before I was looking for a Fedora equivalent of Debian’s debootstrap. The alternatives aren’t really great, so I wrote one instead. Allow me to present febootstrap, a close copy of debootstrap.

One advantage of febootstrap is that it can build new Fedora instances without needing to be root. I use fakeroot and fakechroot to achieve this. These allow you to run commands so that they “think” they’re running as root, when in fact they aren’t. They are frequently used tools in the Debian world, for example, for building and testing packages reproducibly.

$ febootstrap fedora-10 ./f10
$ ls -l ./f10/
total 1096
drwxr-xr-x  2 rjones rjones   4096 2009-03-19 18:38 bin
drwxr-xr-x  2 rjones rjones   4096 2008-09-06 11:13 boot
drwxr-xr-x  4 rjones rjones   4096 2008-09-06 11:13 dev
drwxr-xr-x 14 rjones rjones   4096 2009-03-19 18:38 etc
-rw-rw-r--  1 rjones rjones 231038 2009-03-19 18:38 fakeroot.log
[&c .. note the owner/perms will be correct in the initrd.img]
$ febootstrap-to-initramfs ./f10 > initrd.img

$ febootstrap --groupinstall="Mail Server" fedora-10 ./mailserver
$ febootstrap-to-initramfs ./mailserver > initrd.img

The next step is to work on minimizing those filesystems. At the moment, a “minimal” Fedora install, by which I mean only bash and coreutils and their dependencies, is a shocking 233 MB. If I write that to an initrd image, then that means 233 MB of real RAM is consumed before we even start running the kernel or any programs! Particularly for virtualized environments, this is a real problem. I plan a minimization tool which can strip out anything except the binaries, libraries and other files that are really needed.

(Please note if you really try this, you will certainly need to apply the patch to fakechroot, which can be found in the tarball. That patch is not optional).