Tip: Run Xen as a KVM guest

(Thanks to Dan for pointing out that this is possible)

You can run legacy Xen hypervisor as a KVM guest, which is useful for testing and development. Since the Xen HV won’t have access to the hardware (in particular, to hardware virt) you can only run Xen paravirt guests this way, which in reality means only Linux XenPV guests. [It is supposed to be possible, if you have AMD hardware supporting nested SVN, to get Xen fullvirt guests going, but I did not try this.]


  1. Install a RHEL 5 guest first. This will later be changed into a “Xen host” guest.
  2. You will definitely appreciate having libguestfs around since you can do virt-edit RHEL5Xen /boot/grub/grub.conf to adjust the boot configuration iteratively.

Boot the RHEL 5 guest ordinarily, and install the appropriate kernel-xen and xen packages, and edit grub.conf to enable the Xen hypervisor:

title Red Hat Enterprise Linux Server (2.6.18-194.17.1.el5xen)
        root (hd0,0)
        kernel /xen.gz-2.6.18-194.17.1.el5 noapic
        module /vmlinuz-2.6.18-194.17.1.el5xen ro root=/dev/VolGroup00/LogVol00
        module /initrd-2.6.18-194.17.1.el5xen.img

I added the noapic option, and removed rhgb quiet so I could see what was going on.

Furthermore you need to switch the guest to using emulated devices (IDE, rtl8139) instead of virtio, if it’s not already. Shut down the guest and do:

# virsh edit RHEL5Xen

and change <target dev=’vda’ bus=’virtio’/> to <target dev=’sda’ bus=’ide’/> and remove any <address> element in the same <disk> block. Similarly if there is a network card, make it an emulated rtl8139 instead of virtio.

If during boot you see the error:

request_module: runaway loop modprobe binfmt-464c

then this means you’ve got the wrong kernel-xen package installed, usually that you’ve got the 32 bit kernel-xen.i686 with a 64 bit userspace.

As I said before, I really appreciated guestfish / virt-edit to let me interactively edit the grub configuration and go back and forwards between Xen and ordinary kernel until I got it right.

But finally it did boot, and it was running the Xen hypervisor and dom0:

# /etc/init.d/xend start
Starting xend:                                             [  OK  ]
# /usr/sbin/xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0      864     1 r-----     48.3
# uname -a
Linux rhel5xenx64.home.annexia.org 2.6.18-194.17.1.el5xen #1 SMP Mon Sep 20 07:20:39 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

Update #1

If you have libvirt running inside this guest, it will try to register a bridged network with the address 192.168.122.x which is the exact same address as your host, and that causes confusion. Typical symptoms are that you can’t ssh into the Xen guest from the host, and that networking inside the Xen guest seems broken in strange ways. The solution is simple. In the Xen guest type:

# virsh net-edit default

and change “122” everywhere to something else, eg. “123”.

Then restart the default network:

# virsh net-destroy default
# virsh net-start default

and everything will work again.

Update #2

Because this KVM guest is running a Xen hypervisor and guests, you need to give it a bit more memory. I bumped mine up to 2G, allowing me to comfortably install and run one (nested) guest:

# /usr/sbin/xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0     1224     1 r-----    103.8
RHEL5xenguest                              1      767     1 -b----     33.0


Filed under Uncategorized

4 responses to “Tip: Run Xen as a KVM guest

  1. Christian

    Nice. I came so far, too.
    But I can’t enter the virtual machine and it seems that you didn’t either as the state of the machine says blocked!?

    Any suggestions?



    • rich

      Not sure what you mean by “enter the virtual machine”, but I can log in to both the Xen host and the Xen guest just fine.

      In Xen “blocked” just means the guest is not currently running. It does not mean anything is literally blocked from working.

  2. Simon Easter

    I don’t have the option to disable virtio as outside my control have you been able to find a way around

    udevd timout: killing ‘/sbin/modprobe -v virtio………

    Or is there a way to compile a xen kernel that is able to use virtio block devices?


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.