Composable tools for disk images

Over the past 3 or 4 years, my colleagues and I at Red Hat have been making a set of composable command line tools for handling virtual machine disk images. These let you copy, create, manipulate, display and modify disk images using simple tools that can be connected together in pipelines, while at the same time working very efficiently. It’s all based around the very efficient Network Block Device (NBD) protocol and NBD URI specification.

A basic and very old tool is qemu-img:

$ qemu-img create -f qcow2 disk.qcow2 1G

which creates an empty disk image in qcow2 format. Suppose you want to write into this image? We can compose a few programs:

$ touch disk.raw
$ nbdfuse disk.raw [ qemu-nbd -f qcow2 disk.qcow2 ] &

This serves the qcow2 file up over NBD (qemu-nbd) and then exposes that as a local file using FUSE (nbdfuse). Of interest here, nbdfuse runs and manages qemu-nbd as a subprocess, cleaning it up when the FUSE file is unmounted. We can partition the file using regular tools:

$ gdisk disk.raw
Command (? for help): n
Partition number (1-128, default 1): 
First sector (34-2097118, default = 2048) or {+-}size{KMGTP}: 
Last sector (2048-2097118, default = 2097118) or {+-}size{KMGTP}: 
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 
Changed type of partition to 'Linux filesystem'
Command (? for help): p
Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         2097118   1023.0 MiB  8300  Linux filesystem
Command (? for help): w

Let’s fill that partition with some files using guestfish and unmount it:

$ guestfish -a disk.raw run : \
  mkfs ext2 /dev/sda1 : mount /dev/sda1 / : \
  copy-in ~/libnbd /
$ fusermount3 -u disk.raw
[1]+  Done    nbdfuse disk.raw [ qemu-nbd -f qcow2 disk.qcow2 ]

Now the original qcow2 file is no longer empty but populated with a partition, a filesystem and some files. We can see the space used by examining it with virt-df:

$ virt-df -a disk.qcow2 -h
Filesystem                Size   Used  Available  Use%
disk.qcow2:/dev/sda1     1006M    52M       903M    6%

Now let’s see the first sector. You can’t just “cat” a qcow2 file because it’s a complicated format understood only by qemu. I can assemble qemu-nbd, nbdcopy and hexdump into a pipeline, where qemu-nbd converts the qcow2 format to raw blocks, and nbdcopy copies those out to a pipe:

$ nbdcopy -- [ qemu-nbd -r -f qcow2 disk.qcow2 ] - | \
  hexdump -C -n 512
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001c0  02 00 ee 8a 08 82 01 00  00 00 ff ff 1f 00 00 00  |................|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200

How about instead of a local file, we start with a disk image hosted on a web server, and compressed? We can do that too. Let’s start by querying the size by composing nbdkit’s curl plugin, xz filter and nbdinfo. nbdkit’s --run option composes nbdkit with an external program, connecting them together over an NBD URI ($uri).

$ web=http://mirror.bytemark.co.uk/fedora/linux/development/rawhide/Cloud/x86_64/images/Fedora-Cloud-Base-Rawhide-20220127.n.0.x86_64.raw.xz
$ nbdkit curl --filter=xz $web --run 'nbdinfo $uri'
protocol: newstyle-fixed without TLS
export="":
	export-size: 5368709120 (5G)
	content: DOS/MBR boot sector, extended partition table (last)
	uri: nbd://localhost:10809/
...

Notice it prints the uncompressed (raw) size. Fedora already provides a qcow2 equivalent, but we can also make our own by composing nbdkit, curl, xz, nbdcopy and qemu-nbd:

$ qemu-img create -f qcow2 cloud.qcow2 5368709120 -o preallocation=metadata
$ nbdkit curl --filter=xz $web \
    --run 'nbdcopy -p -- $uri [ qemu-nbd -f qcow2 cloud.qcow2 ]'

Why would you do that instead of downloading and uncompressing? In this case it wouldn’t matter much, but in the general case the disk image might be enormous (terabytes) and you don’t have enough local disk space to do it. Assembling tools into pipelines means you don’t need to keep an intermediate local copy at any point.

We can find out what we’ve got in our new image using various tools:

$ qemu-img info cloud.qcow2 
image: cloud.qcow2
file format: qcow2
virtual size: 5 GiB (5368709120 bytes)
disk size: 951 MiB
$ virt-df -a cloud.qcow2  -h
Filesystem              Size       Used  Available  Use%
cloud.qcow2:/dev/sda2   458M        50M       379M   12%
cloud.qcow2:/dev/sda3   100M       9.8M        90M   10%
cloud.qcow2:/dev/sda5   4.4G       311M       3.6G    7%
cloud.qcow2:btrfsvol:/dev/sda5/root
                        4.4G       311M       3.6G    7%
cloud.qcow2:btrfsvol:/dev/sda5/home
                        4.4G       311M       3.6G    7%
cloud.qcow2:btrfsvol:/dev/sda5/root/var/lib/portables
                        4.4G       311M       3.6G    7%
$ virt-cat -a cloud.qcow2 /etc/redhat-release
Fedora release 36 (Rawhide)

If we wanted to play with the guest in a sandbox, we could stand up an in-memory NBD server populated with the cloud image and connect it to qemu using standard NBD URIs:

$ nbdkit memory 10G
$ qemu-img convert cloud.qcow2 nbd://localhost 
$ virt-customize --format=raw -a nbd://localhost \
    --root-password password:123456 
$ qemu-system-x86_64 -machine accel=kvm \
    -cpu host -m 2048 -serial stdio \
    -drive file=nbd://localhost,if=virtio 
...
fedora login: root
Password: 123456

# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sr0     11:0    1 1024M  0 rom  
zram0  251:0    0  1.9G  0 disk [SWAP]
vda    252:0    0   10G  0 disk 
├─vda1 252:1    0    1M  0 part 
├─vda2 252:2    0  500M  0 part /boot
├─vda3 252:3    0  100M  0 part /boot/efi
├─vda4 252:4    0    4M  0 part 
└─vda5 252:5    0  4.4G  0 part /home
                                /

We can even find out what changed between the in-memory copy and the pristine qcow2 version (quite a lot as it happens):

$ virt-diff --format=raw -a nbd://localhost --format=qcow2 -A cloud.qcow2 
- d 0755       2518 /etc
+ d 0755       2502 /etc
# changed: st_size
- - 0644        208 /etc/.updated
- d 0750        108 /etc/audit
+ d 0750         86 /etc/audit
# changed: st_size
- - 0640         84 /etc/audit/audit.rules
- d 0755         36 /etc/issue.d
+ d 0755          0 /etc/issue.d
# changed: st_size
... for several pages ...

In conclusion, we’ve got a couple of ways to serve disk content over NBD, a set of composable tools for copying, creating, displaying and modifying disk content either from local files or over NBD, and a way to pipe disk data between processes and systems.

We use this in virt-v2v which can suck VMs out of VMware to KVM systems, efficiently, in parallel, and without using local disk space for even the largest guest.

nbdkit 1.12

The new stable release of nbdkit, our flexible Network Block Device server, is out. You can read the announcement and release notes here.

The big new features are SSH support, the linuxdisk plugin, writing plugins in Rust, and extents. Extents allows NBD clients to work out which parts of a disk are sparse or zeroes and skip reading them. It was hellishly difficult to write because of the number of obscure corner cases.

Also in this release, are a couple of interesting filters. The rate filter lets you add a bandwidth limit to connections. We will use this in virt-v2v to allow v2v instances to be rate limited (even dynamically). The readahead filter makes sequential copying and scanning of plugins more efficient by prefetching data ahead of time. It is self-configuring and in most cases simply adding the filter into your filter stack is sufficient to get a nice performance boost, assuming your client’s access patterns are mostly sequential.

Does virt-v2v preserve sparseness?

A common question is does virt-v2v preserve sparseness (aka thin provisioning) when you convert the guest from VMware to a KVM-based hypervisor like oVirt/RHV or OpenStack? The very short answer is: “what?”. The medium answer is: The question doesn’t really make sense. For the long answer, read on …

First of all we need to ask what is thin provisioning? For complex and essentially historical reasons when you provision a new guest you have to decide how much maximum disk space you want it to have. Let’s say you choose 10 GB. However because a new guest install might only take, say, 1 GB, you can also decide if you want the whole 10 GB to be preallocated up front or if you want the disk to be thin provisioned or sparse, meaning it’ll only take up 1 GB of host space, but that will increase as the guest creates new files. There are pros and cons to preallocation. Preallocating the full 10 GB obviously takes a lot of extra space (and what if the guest never uses it?), but it can improve guest performance in some circumstances.

That is what happens initially. By the time we come to do a virt-v2v conversion that guest may have been running for years and years. Now what does the disk look like? It doesn’t matter if the disk was initially thin provisioned or fully allocated, what matters is what the guest did during those years.

Did it repeatedly fill up the disk and/or delete those files? – In which case your initially thin provisioned guest could now be fully allocated.

Did it have trimming enabled? Your initially preallocated guest might now have become sparsely allocated.

In any case VMware doesn’t store this initial state, nor does it make it very easy to find out which bits of the disk are actually backed by host storage and which bits are sparse (well, maybe this is possible, but not using the APIs we use when liberating your guests from VMware).

In any case, as I explained in this talk (slides) from a few years ago, virt-v2v tries to avoid copying any unused, zeroed or deleted parts of the disk for efficiency reasons, and so it will always make the disk maximally sparse when copying it (subject to what the target hypervisor does, read on).

When virt-v2v comes to creating the target guest, the default is to create a maximally sparse guest, but there are two ways to change this:

1. You can specify the -oa preallocated option, where virt-v2v will try to ask the target hypervisor to fully preallocate the target disks of the guest.
2. For some hypervisors, especially RHV, your choice of backend storage may mean you have no option but to use preallocated disks (unfortunately I cannot give clear advice here, best to ask a RHV expert).

The basic rule is that when converting guests you need to think about whether you want the guest to be sparse or preallocated after conversion, based on your own performance vs storage criteria. Whether it happened to be thin provisioned when you set it up on VMware years earlier isn’t a relevant issue.

New in virt-v2v: Import from .vmx files

Virt-v2v converts guests from VMware to KVM, installing any necessary virtio drivers to make the guest work optimally on KVM.

A new feature in virt-v2v 1.37.10 is the ability to import VMware guests directly from disk storage. You do this by pointing virt-v2v directly to the guest’s .vmx metadata file (guest disks are found from references within the VMX file):

$ virt-v2v -i vmx /folder/Fedora_20/Fedora_20.vmx -o local -os /var/tmp
[   0.0] Opening the source -i vmx /folder/Fedora_20/Fedora_20.vmx
[   0.0] Creating an overlay to protect the source from being modified
[   0.1] Initializing the target -o local -os /var/tmp
[   0.1] Opening the overlay
[   6.5] Inspecting the overlay
[  14.0] Checking for sufficient free disk space in the guest
[  14.0] Estimating space required on target for each disk
         ...
[  70.8] Creating output metadata
[  70.8] Finishing off

The problem was how to deal with the case where VMware is storing guests on a NAS (Network Attached Storage). Previously we had to go through VMware to read the guests, either over https or using the proprietary ovftool, but both methods are really slow. If we can directly mount the NAS on the conversion server and read the storage, VMware is no longer involved and things go (much) faster. The result is we’re liberating people from proprietary software much more efficiently.

You’re supposed to use VMware’s proprietary libraries to read and write the VMX file format, which is of course out of the question for virt-v2v, so this was an interesting voyage into VMware’s undocumented and unspecified VMX file format. Superficially it seems like a simple list of key/value pairs in a text file:

.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "10"
nvram = "Fedora_20.nvram"
pciBridge0.present = "TRUE"
svga.present = "TRUE"
...
scsi0.virtualDev = "pvscsi"
scsi0.present = "TRUE"
sata0.present = "TRUE"
scsi0:0.deviceType = "scsi-hardDisk"
scsi0:0.fileName = "Fedora_20.vmdk"
...
sched.cpu.min = "0"
sched.cpu.shares = "normal"
sched.mem.min = "0"
sched.mem.minSize = "0"
sched.mem.shares = "normal"

But there are some things to catch you out:

• All values are quoted, even booleans, integers and lists. This means you cannot distinguish in the parser between strings, booleans and other types. The code which interprets the value must know the type and do the conversion and deal with failures.
• It’s case insensitive, except when it’s case sensitive. In the example above, all of the keys, and the values "TRUE", "normal" and "scsi-hardDisk" are case insensitive. But the filenames ("Fedora_20.vmdk") are case sensitive. The virt-v2v parser tries to be as case insensitive as possible.
• Keys are arranged into a tree. Adding key.present = "FALSE" causes VMware to ignore all keys at that level and below in the tree.
• Quoting of values is plain strange. I’ve never seen the | (pipe) character used as a quoting symbol before.

Luckily libvirt has a large selection of VMX files from the wild we could use to test against.

CentOS 7.2, with added virt-v2v

CentOS 7.2 is out. New in this release, you can simply install virt-v2v and all the dependencies (including libguestfs-winsupport!) are included*.

* = except for libguestfs-xfs which you might also need for converting RHEL 7 guests.

# yum install virt-v2v
Dependencies Resolved

================================================================================
 Package                         Arch   Version                   Repository
                                                                           Size
================================================================================
Installing:
 virt-v2v                        x86_64 1:1.28.1-1.55.el7.centos  base     12 M
Installing for dependencies:
 attr                            x86_64 2.4.46-12.el7             base     66 k
 augeas-libs                     x86_64 1.4.0-2.el7               base    355 k
 boost-system                    x86_64 1.53.0-25.el7             base     39 k
 boost-thread                    x86_64 1.53.0-25.el7             base     57 k
 bridge-utils                    x86_64 1.5-9.el7                 base     32 k
 bzip2                           x86_64 1.0.6-13.el7              base     52 k
 celt051                         x86_64 0.5.1.3-8.el7             base     53 k
 cryptsetup                      x86_64 1.6.7-1.el7               base    119 k
 cyrus-sasl                      x86_64 2.1.26-19.2.el7           base     88 k
 cyrus-sasl-md5                  x86_64 2.1.26-19.2.el7           base     56 k
 device-mapper-event             x86_64 7:1.02.107-5.el7          base    167 k
 device-mapper-event-libs        x86_64 7:1.02.107-5.el7          base    169 k
 device-mapper-persistent-data   x86_64 0.5.5-1.el7               base    350 k
 dosfstools                      x86_64 3.0.20-9.el7              base    101 k
 flac-libs                       x86_64 1.3.0-5.el7_1             base    169 k
 fuse                            x86_64 2.9.2-6.el7               base     84 k
 fuse-libs                       x86_64 2.9.2-6.el7               base     93 k
 gdisk                           x86_64 0.8.6-5.el7               base    187 k
 genisoimage                     x86_64 1.1.11-23.el7             base    298 k
 glusterfs                       x86_64 3.7.1-16.el7              base    422 k
 glusterfs-api                   x86_64 3.7.1-16.el7              base     60 k
 glusterfs-client-xlators        x86_64 3.7.1-16.el7              base    732 k
 glusterfs-libs                  x86_64 3.7.1-16.el7              base    321 k
 gnutls-dane                     x86_64 3.3.8-12.el7_1.1          base     32 k
 gnutls-utils                    x86_64 3.3.8-12.el7_1.1          base    228 k
 gperftools-libs                 x86_64 2.4-7.el7                 base    272 k
 gsm                             x86_64 1.0.13-11.el7             base     30 k
 gtk2                            x86_64 2.24.28-8.el7             base    3.4 M
 hexedit                         x86_64 1.2.13-5.el7              base     39 k
 hivex                           x86_64 1.3.10-5.7.el7            base    100 k
 icoutils                        x86_64 0.31.0-3.el7              base     82 k
 ipxe-roms-qemu                  noarch 20130517-7.gitc4bce43.el7 base    559 k
 iscsi-initiator-utils           x86_64 6.2.0.873-32.el7          base    418 k
 iscsi-initiator-utils-iscsiuio  x86_64 6.2.0.873-32.el7          base     81 k
 ldns                            x86_64 1.6.16-7.el7              base    473 k
 libaio                          x86_64 0.3.109-13.el7            base     24 k
 libasyncns                      x86_64 0.8-7.el7                 base     26 k
 libcgroup                       x86_64 0.41-8.el7                base     64 k
 libconfig                       x86_64 1.4.9-5.el7               base     59 k
 libguestfs                      x86_64 1:1.28.1-1.55.el7.centos  base    1.7 M
 libguestfs-tools-c              x86_64 1:1.28.1-1.55.el7.centos  base    2.2 M
 libguestfs-winsupport           x86_64 7.2-1.el7                 base    2.1 M
 libibverbs                      x86_64 1.1.8-8.el7               base     56 k
 libicu                          x86_64 50.1.2-15.el7             base    6.9 M
 libiscsi                        x86_64 1.9.0-6.el7               base     60 k
 libogg                          x86_64 2:1.3.0-7.el7             base     24 k
 libosinfo                       x86_64 0.2.12-3.el7              base    256 k
 librados2                       x86_64 1:0.80.7-3.el7            base    1.5 M
 librbd1                         x86_64 1:0.80.7-3.el7            base    350 k
 librdmacm                       x86_64 1.0.21-1.el7              base     64 k
 libreport-filesystem            x86_64 2.1.11-32.el7.centos      base     37 k
 libseccomp                      x86_64 2.2.1-1.el7               base     49 k
 libsndfile                      x86_64 1.0.25-10.el7             base    149 k
 libunwind                       x86_64 1.1-5.el7                 base     56 k
 libusal                         x86_64 1.1.11-23.el7             base    135 k
 libvirt-client                  x86_64 1.2.17-13.el7_2.2         updates 4.3 M
 libvirt-daemon                  x86_64 1.2.17-13.el7_2.2         updates 584 k
 libvirt-daemon-driver-interface x86_64 1.2.17-13.el7_2.2         updates 161 k
 libvirt-daemon-driver-network   x86_64 1.2.17-13.el7_2.2         updates 301 k
 libvirt-daemon-driver-nodedev   x86_64 1.2.17-13.el7_2.2         updates 160 k
 libvirt-daemon-driver-nwfilter  x86_64 1.2.17-13.el7_2.2         updates 184 k
 libvirt-daemon-driver-qemu      x86_64 1.2.17-13.el7_2.2         updates 569 k
 libvirt-daemon-driver-secret    x86_64 1.2.17-13.el7_2.2         updates 154 k
 libvirt-daemon-driver-storage   x86_64 1.2.17-13.el7_2.2         updates 327 k
 libvirt-daemon-kvm              x86_64 1.2.17-13.el7_2.2         updates 117 k
 libvorbis                       x86_64 1:1.3.3-8.el7             base    204 k
 libxslt                         x86_64 1.1.28-5.el7              base    242 k
 lsof                            x86_64 4.87-4.el7                base    331 k
 lvm2                            x86_64 7:2.02.130-5.el7          base    1.0 M
 lvm2-libs                       x86_64 7:2.02.130-5.el7          base    872 k
 lzop                            x86_64 1.03-10.el7               base     54 k
 mailcap                         noarch 2.1.41-2.el7              base     31 k
 make                            x86_64 1:3.82-21.el7             base    420 k
 mdadm                           x86_64 3.3.2-7.el7               base    393 k
 mtools                          x86_64 4.0.18-5.el7              base    203 k
 netcf-libs                      x86_64 0.2.8-1.el7               base     69 k
 netpbm                          x86_64 10.61.02-9.el7            base    180 k
 netpbm-progs                    x86_64 10.61.02-9.el7            base    1.9 M
 nmap-ncat                       x86_64 2:6.40-7.el7              base    201 k
 numad                           x86_64 0.5-14.20140620git.el7    base     34 k
 openssl                         x86_64 1:1.0.1e-51.el7_2.1       updates 711 k
 perl-Business-ISBN              noarch 2.06-2.el7                base     25 k
 perl-Business-ISBN-Data         noarch 20120719.001-2.el7        base     24 k
 perl-Compress-Raw-Bzip2         x86_64 2.061-3.el7               base     32 k
 perl-Compress-Raw-Zlib          x86_64 1:2.061-4.el7             base     57 k
 perl-Digest                     noarch 1.17-245.el7              base     23 k
 perl-Digest-MD5                 x86_64 2.52-3.el7                base     30 k
 perl-Encode-Locale              noarch 1.03-5.el7                base     16 k
 perl-File-Listing               noarch 6.04-7.el7                base     13 k
 perl-HTML-Parser                x86_64 3.71-4.el7                base    115 k
 perl-HTML-Tagset                noarch 3.20-15.el7               base     18 k
 perl-HTTP-Cookies               noarch 6.01-5.el7                base     26 k
 perl-HTTP-Daemon                noarch 6.01-5.el7                base     20 k
 perl-HTTP-Date                  noarch 6.02-8.el7                base     14 k
 perl-HTTP-Message               noarch 6.06-6.el7                base     82 k
 perl-HTTP-Negotiate             noarch 6.01-5.el7                base     17 k
 perl-IO-Compress                noarch 2.061-2.el7               base    260 k
 perl-IO-HTML                    noarch 1.00-2.el7                base     23 k
 perl-IO-Socket-IP               noarch 0.21-4.el7                base     35 k
 perl-IO-Socket-SSL              noarch 1.94-3.el7                base    113 k
 perl-LWP-MediaTypes             noarch 6.02-2.el7                base     24 k
 perl-Net-HTTP                   noarch 6.06-2.el7                base     29 k
 perl-Net-LibIDN                 x86_64 0.12-15.el7               base     28 k
 perl-Net-SSLeay                 x86_64 1.55-3.el7                base    285 k
 perl-TimeDate                   noarch 1:2.30-2.el7              base     52 k
 perl-URI                        noarch 1.60-9.el7                base    106 k
 perl-WWW-RobotRules             noarch 6.02-5.el7                base     18 k
 perl-hivex                      x86_64 1.3.10-5.7.el7            base     41 k
 perl-libwww-perl                noarch 6.05-2.el7                base    205 k
 pulseaudio-libs                 x86_64 6.0-7.el7                 base    576 k
 python-chardet                  noarch 2.2.1-1.el7_1             base    227 k
 python-kitchen                  noarch 1.1.1-5.el7               base    267 k
 qemu-img                        x86_64 10:1.5.3-105.el7_2.1      updates 657 k
 qemu-kvm                        x86_64 10:1.5.3-105.el7_2.1      updates 1.8 M
 qemu-kvm-common                 x86_64 10:1.5.3-105.el7_2.1      updates 390 k
 radvd                           x86_64 1.9.2-9.el7               base     85 k
 rsyslog-mmjsonparse             x86_64 7.4.7-12.el7              base     29 k
 scrub                           x86_64 2.5.2-5.el7               base     40 k
 seabios-bin                     noarch 1.7.5-11.el7              base     74 k
 seavgabios-bin                  noarch 1.7.5-11.el7              base     32 k
 sgabios-bin                     noarch 1:0.20110622svn-4.el7     base    7.1 k
 spice-server                    x86_64 0.12.4-15.el7             base    380 k
 supermin5                       x86_64 5.1.10-1.2.el7            base    574 k
 syslinux                        x86_64 4.05-12.el7               base    990 k
 syslinux-extlinux               x86_64 4.05-12.el7               base    363 k
 unbound-libs                    x86_64 1.4.20-26.el7             base    296 k
 unzip                           x86_64 6.0-15.el7                base    166 k
 usbredir                        x86_64 0.6-7.el7                 base     44 k
 yajl                            x86_64 2.0.4-4.el7               base     39 k
 yum-utils                       noarch 1.1.31-34.el7             base    113 k

Transaction Summary
================================================================================
Install  1 Package (+130 Dependent packages)

Total download size: 59 M
Installed size: 172 M

virt-v2v, libguestfs and qemu remote drivers in RHEL 7

Upstream qemu can access a variety of remote disks, like NBD and Ceph. This feature is exposed in libguestfs so you can easily mount remote storage.

However in RHEL 7 many of these drivers are disabled, because they’re not stable enough to support. I was asked exactly how this works, and this post is my answer — as it’s not as simple as it sounds.

There are (at least) five separate layers involved:

qemu code	What block drivers are compiled into qemu, and which ones are compiled out completely.
qemu block driver r/o whitelist	A whitelist of drivers that qemu allows you to use read-only.
qemu block driver r/w whitelist	A whitelist of drivers that qemu allows you to use for read and write.
libvirt	What libvirt enables (not covered in this discussion).
libguestfs	In RHEL we patch out some qemu remote storage types using a custom patch.

Starting at the bottom of the stack, in RHEL we use ./configure --disable-* flags to disable a few features: Ceph is disabled on !x86_64 and 9pfs is disabled everywhere. This means the qemu binary won’t even contain code for those features.

If you run qemu-img --help in RHEL 7, you’ll see the drivers which are compiled into the binary:

$ rpm -qf /usr/bin/qemu-img
qemu-img-1.5.3-92.el7.x86_64
$ qemu-img --help
[...]
Supported formats: vvfat vpc vmdk vhdx vdi ssh
sheepdog rbd raw host_cdrom host_floppy host_device
file qed qcow2 qcow parallels nbd iscsi gluster dmg
tftp ftps ftp https http cloop bochs blkverify blkdebug

Although you can use all of those in qemu-img, not all of those drivers work in qemu (the hypervisor). qemu implements two whitelists. The RHEL 7 qemu-kvm.spec file looks like this:

./configure [...]
    --block-drv-rw-whitelist=qcow2,raw,file,host_device,blkdebug,nbd,iscsi,gluster,rbd \
    --block-drv-ro-whitelist=vmdk,vhdx,vpc,ssh,https

The --block-drv-rw-whitelist parameter configures the drivers for which full read and write access is permitted and supported in RHEL 7. It’s quite a short list!

Even shorter is the --block-drv-ro-whitelist parameter — drivers for which only read-only access is allowed. You can’t use qemu to open these files for write. You can use these as (read-only) backing files, but you can’t commit to those backing files.

In practice what happens is you get an error if you try to use non-whitelisted block drivers:

$ /usr/libexec/qemu-kvm -drive file=test.vpc
qemu-kvm: -drive file=test.vpc: could not open disk image
test.vpc: Driver 'vpc' can only be used for read-only devices
$ /usr/libexec/qemu-kvm -drive file=test.qcow1
qemu-kvm: -drive file=test.qcow1: could not open disk
image test.qcow1: Driver 'qcow' is not whitelisted

Note that’s a qcow v1 (ancient format) file, not modern qcow2.

Side note: Only qemu (the hypervisor) enforces the whitelist. Tools like qemu-img ignore it.

At the top of the stack, libguestfs has a patch which removes support for many remote protocols. Currently (RHEL 7.2/7.3) we disable: ftp, ftps, http, https, tftp, gluster, iscsi, sheepdog, ssh. That leaves only: local file, rbd (Ceph) and NBD enabled.

virt-v2v uses a mixture of libguestfs and qemu-img to convert VMware and Xen guests to run on KVM. To access VMware we need to use https and to access Xen we use ssh. Both of those drivers are disabled in libguestfs, and only available read-only in the qemu whitelist. However that’s sufficient for virt-v2v, since all it does is add the https or ssh driver as a read-only backing file. (If you are interested in finding out more about how virt-v2v works, then I gave a talk about it at the KVM Forum which is available online).

In summary — it’s complicated.

Importing KVM guests to oVirt or RHEV

One of the tools I maintain is virt-v2v. It’s a program to import guests from foreign hypervisors like VMware and Xen, to KVM. It only does conversions to KVM, not the other way. And a feature I intentionally removed in RHEL 7 was importing KVM → KVM.

Why would you want to “import” KVM → KVM? Well, no reason actually. In fact it’s one of those really bad ideas for V2V. However it used to have a useful purpose: oVirt/RHEV can’t import a plain disk image, but virt-v2v knows how to import things to oVirt, so people used virt-v2v as backdoor for this missing feature.

Removing this virt-v2v feature has caused a lot of moaning, but I’m adamant it’s a very bad idea to use virt-v2v as a way to import disk images. Virt-v2v does all sorts of complex filesystem and Windows Registry manipulations, which you don’t want and don’t need if your guest already runs on KVM. Worst case, you could even end up breaking your guest.

However I have now written a replacement script that does the job: http://git.annexia.org/?p=import-to-ovirt.git

If your guest is a disk image that already runs on KVM, then you can use this script to import the guest. You’ll need to clone the git repo, read the README file, and then read the tool’s man page. It’s pretty straightforward.

There are a few shortcomings with this script to be aware of:

1. The guest must have virtio drivers installed already, and must be able to boot off virtio-blk (default) or virtio-scsi. For virtio-scsi, you’ll need to flip the checkbox in the ‘Advanced’ section of the guest parameters in the oVirt UI.
2. It should be possible to import guests that don’t have virtio drivers installed, but can use IDE. This is a missing feature (patches welcome).
3. No network card is added to the guest, so it probably won’t have network when it boots. It should be possible to add a network card through the UI, but really this is something that needs to be fixed in the script (patches welcome).
4. It doesn’t handle all the random packaging formats that guests come in, like OVA. You’ll have to extract these first and import just the disk image.
5. It’s not in any way supported or endorsed by Red Hat.

How to rebuild libguestfs from source on RHEL or CentOS 7

Three people have asked me about this, so here goes. You will need a RHEL or CentOS 7.1 machine (perhaps a VM), and you may need to grab extra packages from this preview repository. The preview repo will go away when we release 7.2, but then again 7.2 should contain all the packages you need.

You’ll need to install rpm-build. You could also install mock (from EPEL), but in fact you don’t need mock to build libguestfs and it may be easier and faster without.

Please don’t build libguestfs as root. It’s not necessary to build (any) packages as root, and can even be dangerous.

Grab the source RPM. The latest at time of writing is libguestfs-1.28.1-1.55.el7.src.rpm. When 7.2 comes out, you’ll be able to get the source RPM using this command:

yumdownloader --source libguestfs

I find it helpful to build RPMs in my home directory, and also to disable the libguestfs tests. To do that, I have a ~/.rpmmacros file that contains:

%_topdir	%(echo $HOME)/rpmbuild
%_smp_mflags	-j5
%libguestfs_runtests   0

You may wish to adjust %_smp_mflags. A good value to choose is 1 + the number of cores on your machine.

I’ll assume at this point that the reason you want to rebuild libguestfs is to apply a patch (otherwise why aren’t you using the binaries we supply?), so first let’s unpack the source tree. Note I am running this command as non-root:

rpm -i libguestfs-1.28.1-1.55.el7.src.rpm

If you set up ~/.rpmmacros as above then the sources should be unpacked under ~/rpmbuild/SPECS and ~/rpmbuild/SOURCES.

Take a look at least at the libguestfs.spec file. You may wish to modify it now to add any patches you need (add the patch files to the SOURCES/ subdirectory). You might also want to modify the Release: tag so that your package doesn’t conflict with the official package.

You might also need to install build dependencies. This command should be run as root since it needs to install packages, and also note that you may need packages from the repo linked above.

yum-builddep libguestfs.spec

Now you can rebuild libguestfs (non-root!):

rpmbuild -ba libguestfs.spec

With the tests disabled, on decent hardware, that should take about 10 minutes.

The final binary packages will end up in ~/rpmbuild/RPMS/ and can be installed as normal:

yum localupdate x86_64/*.rpm noarch/*.rpm

You might see errors during the build phase. If they aren’t fatal, you can ignore them, but if the build fails then post the complete log to our mailing list (you don’t need to subscribe) so we can help you out.

My KVM Forum 2015 talk: New qemu technology used in virt-v2v

All KVM Forum talks can be found here.

KVM Forum 2015

Assuming HMG can get my passport back to me in time, I am speaking at the KVM Forum 2015 in Seattle USA (full schedule of talks here).

I’m going to be talking about virt-v2v and new features of qemu/KVM that made it possible for virt-v2v to be faster and more reliable than ever.