Tag Archives: libvirt

KVM working on the Cubietruck


I managed to get KVM working on the Cubietruck last week. It’s not exactly simple, but this post describes in overview how to do it.

(1) You will need a Cubietruck, a CP2102 serial cable, a micro SDHC card, a card reader for your host computer, and a network patch cable (the board supports wifi but it doesn’t work with the newer kernel we’ll be using). Optional: 2.5″ SATA HDD or SSD.

(2) Start with Hans De Goede’s AllWinner remix of Fedora 19, and get that working. It’s important to read his README file carefully.

(3) Build this upstream kernel with this configuration:

make oldconfig
make menuconfig

In menuconfig, enable Large Page Address Extension (LPAE), and then enable KVM in the Virtualization menu.

LOADADDR=0x40008000 make uImage dtbs
make modules
sudo cp arch/arm/boot/uImage /boot/uImage.sunxi-test
sudo cp arch/arm/boot/dts/sun7i-a20-cubietruck.dtb /boot/sun7i-a20-cubietruck.dtb.sunxi-test
sudo make modules_install

Reboot, interrupt u-boot (using the serial console), and type the following commands to load the new kernel:

setenv bootargs console=ttyS0,115200 loglevel=9 earlyprintk ro rootwait root=/dev/mmcblk0p3
ext2load mmc 0 0x46000000 uImage.sunxi-test
ext2load mmc 0 0x4b000000 sun7i-a20-cubietruck.dtb.sunxi-test
env set fdt_high ffffffff
bootm 0x46000000 - 0x4b000000

(4) Build this modified u-boot which supports Hyp mode.

make cubietruck_config
sudo dd if=u-boot-sunxi-with-spl.bin of=/dev/YOURSDCARD bs=1024 seek=8

Reboot again, use the commands above to boot into the upstream kernel, and if everything worked you should see:

Brought up 2 CPUs
SMP: Total of 2 processors activated.
CPU: All CPU(s) started in HYP mode.
CPU: Virtualization extensions available.

Also /dev/kvm should exist.

(5) Hack QEMU to create Cortex-A7 CPUs using this one-line patch.

Edit: dgilmore tells me this is no longer necessary. Instead make sure you use the qemu -cpu host option.

Then you should be able to create VMs using libvirt. Note if using libguestfs you will need to use the direct backend (LIBGUESTFS_BACKEND=direct) because of this libvirt bug.


Filed under Uncategorized

Creating a cloud-init config disk for non-cloud boots

There are lots of cloud disk images floating around. They are designed to run in clouds where there is a boot-time network service called cloud-init available that provides initial configuration. If that’s not present, or you’re just trying to boot these images in KVM/libvirt directly without any cloud, then things can go wrong.

Luckily it’s fairly easy to create a config disk (aka “seed disk”) which you attach to the guest and then let cloud-init in the guest get its configuration from there. No cloud, or even network, required.

I’m going to use a tool called virt-make-fs to make the config disk, as it’s easy to use and doesn’t require root. There are other tools around, eg. make-seed-disk which do a similar job. (NB: You might hit this bug in virt-make-fs, which should be fixed in the latest version).

I’m also using a cloud image downloaded from the Fedora project, but any cloud image should work.

First I create my cloud-init metadata. This consists of two files. meta-data contains host and network configuration:

instance-id: iid-123456
local-hostname: cloudy

user-data contains other custom configuration (note #cloud-config is
not a comment, it’s a directive to tell cloud-init the format of the file):

password: 123456
 - [ useradd, -m, -p, "", rjones ]
 - [ chage, -d, 0, rjones ]

(The idea behind this split is probably not obvious, but apparently it’s because the meta-data is meant to be supplied by the Cloud, and the user-data is meant to be supplied by the Cloud’s customer. In this case, no cloud, so we’re going to supply both!)

I put these two files into a directory, and run virt-make-fs to create the config disk:

$ ls
meta-data  user-data
$ virt-make-fs --type=msdos --label=cidata . /tmp/seed.img
$ virt-filesystems -a /tmp/seed.img --all --long -h
Name      Type        VFS   Label   MBR  Size  Parent
/dev/sda  filesystem  vfat  cidata  -    286K  -
/dev/sda  device      -     -       -    286K  -

Now I need to pass some kernel options when booting the Fedora cloud image, and the only way to do that is if I boot from an external kernel & initrd. This is not as complicated as it sounds, and virt-builder has an option to get the kernel and initrd that I’m going to need:

$ virt-builder --get-kernel Fedora-cloud.raw
download: /boot/vmlinuz-3.9.5-301.fc19.x86_64 -> ./vmlinuz-3.9.5-301.fc19.x86_64
download: /boot/initramfs-3.9.5-301.fc19.x86_64.img -> ./initramfs-3.9.5-301.fc19.x86_64.img

Finally I’m going to boot the guest using KVM (you could also use libvirt with a little extra effort):

$ qemu-kvm -m 1024 \
    -drive file=Fedora-cloud.raw,if=virtio \
    -drive file=seed.img,if=virtio \
    -kernel ./vmlinuz-3.9.5-301.fc19.x86_64 \
    -initrd ./initramfs-3.9.5-301.fc19.x86_64.img \
    -append 'root=/dev/vda1 ro ds=nocloud-net'

You’ll be able to log in either as fedora/123456 or rjones (no password), and you should see that the hostname has been set to cloudy.


Filed under Uncategorized

New tool: virt-builder

New in libguestfs 1.24 will be a simple tool called virt-builder. This builds virtual machines of various free operating systems quickly and securely:

$ virt-builder fedora-19 --size 20G --install nmap
[     0.0] Downloading: http://libguestfs.org/download/builder/fedora-19.xz
[     2.0] Uncompressing: http://libguestfs.org/download/builder/fedora-19.xz
[    25.0] Running virt-resize to expand the disk to 20.0G
[    74.0] Opening the new disk
[    78.0] Random root password: RCuMKJ4NPak0ptJQ [did you mean to use --root-password?]
[    78.0] Installing packages: nmap
[    93.0] Finishing off

Some notable features:

  • Fast: As you can see above, once it has downloaded and cached the template first time, it can churn out new guests in around 90 seconds.
  • Install packages.
  • Set the hostname.
  • Generate a random seed for the guest.
  • Upload files.
  • Set passwords, create user accounts.
  • Run custom scripts.
  • Install firstboot scripts.
  • Fetch packages from private repos and ISOs.
  • Secure: Everything is assembled in a container (using SELinux if available).
  • Guest templates are PGP-signed.
  • No root or privileged access needed at all (no setuid, no sudo).
  • Fully scriptable.
  • Can be used in locked-down no-network scenarios.
  • Can use UML as a backend (good for use in a cloud).


Filed under Uncategorized

An ARM libvirt guest (on an x86-64 host)

Cole Robinson has fixed libvirt sufficiently that you can now run ARM libvirt guests on x86-64 hosts.

You’ll need the virtualization packages from Fedora 20. If you have Fedora 19, then you can recompile the following packages from Fedora Rawhide: libvirt, qemu, SLOF, virt-manager, libssh2, qemu-sanity-check. Or use Cole’s virt-preview repo. Then just follow Cole’s instructions.

$ uname -a
Linux trick 3.10.9-200.fc19.x86_64 #1 SMP
Wed Aug 21 19:27:58 UTC 2013 x86_64 x86_64 x86_64
$ sudo virsh list
 Id    Name                           State
 9     f19arm                         running

$ ssh
Last login: Tue Sep  3 11:43:27 2013 from
$ uname -a
Linux localhost 3.9.5-301.fc19.armv7hl #1 SMP
Wed Jun 12 14:56:17 UTC 2013 armv7l armv7l armv7l

1 Comment

Filed under Uncategorized

The Facebook Platform

“The rapidness of web distribution has made older development practices seem quaint and antiquated. But something that’s not antiquated, or shouldn’t be, is providing a service that does what claims, that provides more value than it takes back, and that earnestly cares about the way it gets used”

I don’t often link to other blog posts, but this one is excellent. The Facebook Platform sounds a lot like the Google AdWords API that I used to have the misfortune to write software for. A sociopathic API, constantly changing, that didn’t care about the developers that had to use it. How many of those exist also in the open source world.

I think the best thing libvirt did was to offer a stable API and ABI from day 1. Programs written to a published API should rarely (ideally, never) need to be changed.

Leave a comment

Filed under Uncategorized

Reminder: I’m speaking next week

At the CentOS Dojo in Aldershot, on Friday 12th July. Tickets are £15 per person (don’t worry, that’s not just for me, there are lots of other speakers!), and there’s beer and pig roast in the evening included in the price.

Ask me anything you want about virtualization. I might even be able to answer.

Leave a comment

Filed under Uncategorized

New project: nbdkit, liberally licensed NBD server with a plugin API

Last week I started a new project: nbdkit. This is a toolkit for creating NBD servers. The key features are:

  1. Multithreaded NBD server written in C with good performance.
  2. Well-documented, simple plugin API with a stable ABI guarantee. Let’s you export “unconventional” block devices easily.
  3. Liberal license (BSD) allows nbdkit to be linked to proprietary libraries or included in proprietary code.

There are of course many NBD servers already, such as the original nbd project, qemu-nbd and jnbds.

There are also a handful of servers specialized for particular disk sources. A good example of that is this OpenStack Swift server. But you shouldn’t have to write a whole new server just to export a new disk type.

nbdkit hopefully offers a unique contribution to this field because it’s a general server with a plugin architecture, offering a stable ABI and a liberal license so you can link it to proprietary code (say hello, VDDK).

The motivation for this is to make many more data sources available to libguestfs. Especially I want to write plugins for libvirt, VDDK and some OpenStack sources.


Filed under Uncategorized