Tip: Run Xen as a KVM guest

(Thanks to Dan for pointing out that this is possible)

You can run legacy Xen hypervisor as a KVM guest, which is useful for testing and development. Since the Xen HV won’t have access to the hardware (in particular, to hardware virt) you can only run Xen paravirt guests this way, which in reality means only Linux XenPV guests. [It is supposed to be possible, if you have AMD hardware supporting nested SVN, to get Xen fullvirt guests going, but I did not try this.]

Preparation:

1. Install a RHEL 5 guest first. This will later be changed into a “Xen host” guest.
2. You will definitely appreciate having libguestfs around since you can do virt-edit RHEL5Xen /boot/grub/grub.conf to adjust the boot configuration iteratively.

Boot the RHEL 5 guest ordinarily, and install the appropriate kernel-xen and xen packages, and edit grub.conf to enable the Xen hypervisor:

default=0
[...]
title Red Hat Enterprise Linux Server (2.6.18-194.17.1.el5xen)
        root (hd0,0)
        kernel /xen.gz-2.6.18-194.17.1.el5 noapic
        module /vmlinuz-2.6.18-194.17.1.el5xen ro root=/dev/VolGroup00/LogVol00
        module /initrd-2.6.18-194.17.1.el5xen.img

I added the noapic option, and removed rhgb quiet so I could see what was going on.

Furthermore you need to switch the guest to using emulated devices (IDE, rtl8139) instead of virtio, if it’s not already. Shut down the guest and do:

# virsh edit RHEL5Xen

and change <target dev=’vda’ bus=’virtio’/> to <target dev=’sda’ bus=’ide’/> and remove any <address> element in the same <disk> block. Similarly if there is a network card, make it an emulated rtl8139 instead of virtio.

If during boot you see the error:

request_module: runaway loop modprobe binfmt-464c

then this means you’ve got the wrong kernel-xen package installed, usually that you’ve got the 32 bit kernel-xen.i686 with a 64 bit userspace.

As I said before, I really appreciated guestfish / virt-edit to let me interactively edit the grub configuration and go back and forwards between Xen and ordinary kernel until I got it right.

But finally it did boot, and it was running the Xen hypervisor and dom0:

# /etc/init.d/xend start
Starting xend:                                             [  OK  ]
# /usr/sbin/xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0      864     1 r-----     48.3
# uname -a
Linux rhel5xenx64.home.annexia.org 2.6.18-194.17.1.el5xen #1 SMP Mon Sep 20 07:20:39 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

Update #1

If you have libvirt running inside this guest, it will try to register a bridged network with the address 192.168.122.x which is the exact same address as your host, and that causes confusion. Typical symptoms are that you can’t ssh into the Xen guest from the host, and that networking inside the Xen guest seems broken in strange ways. The solution is simple. In the Xen guest type:

# virsh net-edit default

and change “122” everywhere to something else, eg. “123”.

Then restart the default network:

# virsh net-destroy default
# virsh net-start default

and everything will work again.

Update #2

Because this KVM guest is running a Xen hypervisor and guests, you need to give it a bit more memory. I bumped mine up to 2G, allowing me to comfortably install and run one (nested) guest:

# /usr/sbin/xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0     1224     1 r-----    103.8
RHEL5xenguest                              1      767     1 -b----     33.0