New in virt-v2v: Import from .vmx files

Virt-v2v converts guests from VMware to KVM, installing any necessary virtio drivers to make the guest work optimally on KVM.

A new feature in virt-v2v 1.37.10 is the ability to import VMware guests directly from disk storage. You do this by pointing virt-v2v directly to the guest’s .vmx metadata file (guest disks are found from references within the VMX file):

$ virt-v2v -i vmx /folder/Fedora_20/Fedora_20.vmx -o local -os /var/tmp
[   0.0] Opening the source -i vmx /folder/Fedora_20/Fedora_20.vmx
[   0.0] Creating an overlay to protect the source from being modified
[   0.1] Initializing the target -o local -os /var/tmp
[   0.1] Opening the overlay
[   6.5] Inspecting the overlay
[  14.0] Checking for sufficient free disk space in the guest
[  14.0] Estimating space required on target for each disk
         ...
[  70.8] Creating output metadata
[  70.8] Finishing off

The problem was how to deal with the case where VMware is storing guests on a NAS (Network Attached Storage). Previously we had to go through VMware to read the guests, either over https or using the proprietary ovftool, but both methods are really slow. If we can directly mount the NAS on the conversion server and read the storage, VMware is no longer involved and things go (much) faster. The result is we’re liberating people from proprietary software much more efficiently.

You’re supposed to use VMware’s proprietary libraries to read and write the VMX file format, which is of course out of the question for virt-v2v, so this was an interesting voyage into VMware’s undocumented and unspecified VMX file format. Superficially it seems like a simple list of key/value pairs in a text file:

.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "10"
nvram = "Fedora_20.nvram"
pciBridge0.present = "TRUE"
svga.present = "TRUE"
...
scsi0.virtualDev = "pvscsi"
scsi0.present = "TRUE"
sata0.present = "TRUE"
scsi0:0.deviceType = "scsi-hardDisk"
scsi0:0.fileName = "Fedora_20.vmdk"
...
sched.cpu.min = "0"
sched.cpu.shares = "normal"
sched.mem.min = "0"
sched.mem.minSize = "0"
sched.mem.shares = "normal"

But there are some things to catch you out:

• All values are quoted, even booleans, integers and lists. This means you cannot distinguish in the parser between strings, booleans and other types. The code which interprets the value must know the type and do the conversion and deal with failures.
• It’s case insensitive, except when it’s case sensitive. In the example above, all of the keys, and the values "TRUE", "normal" and "scsi-hardDisk" are case insensitive. But the filenames ("Fedora_20.vmdk") are case sensitive. The virt-v2v parser tries to be as case insensitive as possible.
• Keys are arranged into a tree. Adding key.present = "FALSE" causes VMware to ignore all keys at that level and below in the tree.
• Quoting of values is plain strange. I’ve never seen the | (pipe) character used as a quoting symbol before.

Luckily libvirt has a large selection of VMX files from the wild we could use to test against.

virt-v2v, libguestfs and qemu remote drivers in RHEL 7

Upstream qemu can access a variety of remote disks, like NBD and Ceph. This feature is exposed in libguestfs so you can easily mount remote storage.

However in RHEL 7 many of these drivers are disabled, because they’re not stable enough to support. I was asked exactly how this works, and this post is my answer — as it’s not as simple as it sounds.

There are (at least) five separate layers involved:

qemu code	What block drivers are compiled into qemu, and which ones are compiled out completely.
qemu block driver r/o whitelist	A whitelist of drivers that qemu allows you to use read-only.
qemu block driver r/w whitelist	A whitelist of drivers that qemu allows you to use for read and write.
libvirt	What libvirt enables (not covered in this discussion).
libguestfs	In RHEL we patch out some qemu remote storage types using a custom patch.

Starting at the bottom of the stack, in RHEL we use ./configure --disable-* flags to disable a few features: Ceph is disabled on !x86_64 and 9pfs is disabled everywhere. This means the qemu binary won’t even contain code for those features.

If you run qemu-img --help in RHEL 7, you’ll see the drivers which are compiled into the binary:

$ rpm -qf /usr/bin/qemu-img
qemu-img-1.5.3-92.el7.x86_64
$ qemu-img --help
[...]
Supported formats: vvfat vpc vmdk vhdx vdi ssh
sheepdog rbd raw host_cdrom host_floppy host_device
file qed qcow2 qcow parallels nbd iscsi gluster dmg
tftp ftps ftp https http cloop bochs blkverify blkdebug

Although you can use all of those in qemu-img, not all of those drivers work in qemu (the hypervisor). qemu implements two whitelists. The RHEL 7 qemu-kvm.spec file looks like this:

./configure [...]
    --block-drv-rw-whitelist=qcow2,raw,file,host_device,blkdebug,nbd,iscsi,gluster,rbd \
    --block-drv-ro-whitelist=vmdk,vhdx,vpc,ssh,https

The --block-drv-rw-whitelist parameter configures the drivers for which full read and write access is permitted and supported in RHEL 7. It’s quite a short list!

Even shorter is the --block-drv-ro-whitelist parameter — drivers for which only read-only access is allowed. You can’t use qemu to open these files for write. You can use these as (read-only) backing files, but you can’t commit to those backing files.

In practice what happens is you get an error if you try to use non-whitelisted block drivers:

$ /usr/libexec/qemu-kvm -drive file=test.vpc
qemu-kvm: -drive file=test.vpc: could not open disk image
test.vpc: Driver 'vpc' can only be used for read-only devices
$ /usr/libexec/qemu-kvm -drive file=test.qcow1
qemu-kvm: -drive file=test.qcow1: could not open disk
image test.qcow1: Driver 'qcow' is not whitelisted

Note that’s a qcow v1 (ancient format) file, not modern qcow2.

Side note: Only qemu (the hypervisor) enforces the whitelist. Tools like qemu-img ignore it.

At the top of the stack, libguestfs has a patch which removes support for many remote protocols. Currently (RHEL 7.2/7.3) we disable: ftp, ftps, http, https, tftp, gluster, iscsi, sheepdog, ssh. That leaves only: local file, rbd (Ceph) and NBD enabled.

virt-v2v uses a mixture of libguestfs and qemu-img to convert VMware and Xen guests to run on KVM. To access VMware we need to use https and to access Xen we use ssh. Both of those drivers are disabled in libguestfs, and only available read-only in the qemu whitelist. However that’s sufficient for virt-v2v, since all it does is add the https or ssh driver as a read-only backing file. (If you are interested in finding out more about how virt-v2v works, then I gave a talk about it at the KVM Forum which is available online).

In summary — it’s complicated.

How to rebuild libguestfs from source on RHEL or CentOS 7

Three people have asked me about this, so here goes. You will need a RHEL or CentOS 7.1 machine (perhaps a VM), and you may need to grab extra packages from this preview repository. The preview repo will go away when we release 7.2, but then again 7.2 should contain all the packages you need.

You’ll need to install rpm-build. You could also install mock (from EPEL), but in fact you don’t need mock to build libguestfs and it may be easier and faster without.

Please don’t build libguestfs as root. It’s not necessary to build (any) packages as root, and can even be dangerous.

Grab the source RPM. The latest at time of writing is libguestfs-1.28.1-1.55.el7.src.rpm. When 7.2 comes out, you’ll be able to get the source RPM using this command:

yumdownloader --source libguestfs

I find it helpful to build RPMs in my home directory, and also to disable the libguestfs tests. To do that, I have a ~/.rpmmacros file that contains:

%_topdir	%(echo $HOME)/rpmbuild
%_smp_mflags	-j5
%libguestfs_runtests   0

You may wish to adjust %_smp_mflags. A good value to choose is 1 + the number of cores on your machine.

I’ll assume at this point that the reason you want to rebuild libguestfs is to apply a patch (otherwise why aren’t you using the binaries we supply?), so first let’s unpack the source tree. Note I am running this command as non-root:

rpm -i libguestfs-1.28.1-1.55.el7.src.rpm

If you set up ~/.rpmmacros as above then the sources should be unpacked under ~/rpmbuild/SPECS and ~/rpmbuild/SOURCES.

Take a look at least at the libguestfs.spec file. You may wish to modify it now to add any patches you need (add the patch files to the SOURCES/ subdirectory). You might also want to modify the Release: tag so that your package doesn’t conflict with the official package.

You might also need to install build dependencies. This command should be run as root since it needs to install packages, and also note that you may need packages from the repo linked above.

yum-builddep libguestfs.spec

Now you can rebuild libguestfs (non-root!):

rpmbuild -ba libguestfs.spec

With the tests disabled, on decent hardware, that should take about 10 minutes.

The final binary packages will end up in ~/rpmbuild/RPMS/ and can be installed as normal:

yum localupdate x86_64/*.rpm noarch/*.rpm

You might see errors during the build phase. If they aren’t fatal, you can ignore them, but if the build fails then post the complete log to our mailing list (you don’t need to subscribe) so we can help you out.

KVM Forum 2015

Assuming HMG can get my passport back to me in time, I am speaking at the KVM Forum 2015 in Seattle USA (full schedule of talks here).

I’m going to be talking about virt-v2v and new features of qemu/KVM that made it possible for virt-v2v to be faster and more reliable than ever.

Video: virt-v2v integration with RHEV-M

This video shows using the GUI to import a virtual machine from VMware to RHEV-M. It performs the conversion using virt-v2v, which is responsible for installing virtio drivers, fixing the bootloader, and so forth.

Thanks Arik Hadas. Now I just have to fix the epic RHEL 7.2 bug list — 57 bugs at last count 😦

libguestfs 1.28 released

The new stable version of libguestfs — a C library and tools for accessing and modifying virtual machine disk images — has been released.

There is one brand new tool, virt-log. And I rewrote the virt-v2v and virt-p2v tools. These tools convert VMware and Xen guests and physical machines, to run on KVM. They are now much faster and better than before.

As well as that there are hundreds of other improvements and bug fixes. For a full list, see the release notes.

Libguestfs 1.28 will be available shortly in Fedora 21, Debian/experimental, RHEL and CentOS 7, and elsewhere.

P2V and V2V: new release

Matt has just released virt-p2v and virt-v2v 0.8.5. Packages will be available in Fedora 16 updates-testing shortly. Read this for the full instructions:

https://www.redhat.com/archives/libguestfs/2011-December/msg00061.html

Red Hat Summit video online

I’m working to get this in a more usable format, but at the moment if you have flash you can watch Matt and I presenting virt-v2v and libguestfs at the URL below.

Click “Red Hat Summit”, then “V2V VMWare/Xen”.

Matt does the first 30 minutes, then I take over at almost exactly the 30’00 mark in the video.

http://www.redhat.com/promo/summit/2010/highlights/

Outline for V2V session at the Red Hat Summit 2010

Posted here …

“In this session, Rich Jones and Matt Booth, senior software engineers at Red Hat, will introduce the virtualization V2V tool. They will also demonstrate how users can easily convert VMware and Xen virtual machines from the native format into KVM images for use in Red Hat Enterprise Linux and Red Hat Enterprise Virtualization. Rich and Matt will detail the step-by-step instructions for converting existing virtual machines, and provide an in-depth look into the technologies, including libguestfs, used to build the V2V tool.”

What things make P2V/V2V conversion hard?

(In case anyone is confused by the title, P2V means turning a physical system into a virtual machine, and V2V means converting one type of virtual machine into another — eg. a Xen VM into a VMWare VM. Usually what we are talking about is making the conversion completely or mostly automatic.)

There are some things that Linux distros do which make P2V and V2V conversions harder than they really need to be. In this article I hope to collect a few of these things. If you can think of any more, post them as comments and I’ll incorporate them here.

1. Your disk partitions are on hard-coded device names like /dev/hda1

When we virtualize a disk, we usually want to install alternate (“paravirt”) drivers, but these drivers cause the disk names to change. Xen paravirt drivers use /dev/xvda and the standard Linux virtio drivers use /dev/vda.

Your Linux distro may have scattered references to /dev/hd* and /dev/sd* in /etc/fstab, inside the boot initramfs, and maybe in some scripts which are used to decrypt the hard drive at boot time. virt-v2v has to perform intimate surgery on the disk to find and fix all these.

Luckily there is a easy way to avoid this: every major filesystem and swap type for Linux supports either labels or UUIDs. These are preserved transparently when converting, and distros should use these scrupulously, and never use device names.

Update: LVM partition names are OK too. (Thanks Matt Booth).

2. You didn’t expect the network hardware would change

When the OS gets converted, it is highly likely that the network device will appear to change, but mysteriously it will still have the same MAC address.

That’s not supposed to happen with real hardware — MAC addrs are allocated by IEEE and no two manufacturers should have the same ones. Virtual hardware doesn’t play by the same rules.

What we’d want you to do is to keep track of the MAC addresses of each interface and when you see the same MAC, give it the same ethX device name.

Update: See comments.

3. Your kernel doesn’t support virtual devices

Some Linux distros don’t contain drivers for virtual (virtio) devices. There’s a variety of reasons for this, not all of them solvable: The kernel might predate virtio, or the drivers are closed source (eg. for older VMWare), or they might have been compiled out. In some cases a whole different kernel is required (as was true for earlier versions of Xen).

In any case, installing a new kernel inside the guest and making it bootable is major surgery, and we’d rather not do that.

4. X needs reconfiguration

Pretty much the same as for disks and network devices, the display device will also change after conversion (either to a generic Cirrus Logic 54xx or to one of the new accelerated paravirt devices).

Ideally X would just deal with this and bring up a display on the first device it finds, and it should probe all possible devices, but by no means all distros work this way.

5. Making assumptions about the environment around the machine

Static IP addresses and static DNS resolution are a pain when moving a machine. Although not necessarily the fault of the distro as such, it’s better if machines are configured to pick up their network details from a DHCP server.

6. Don’t assume CPU extensions like SSE3 will always be there

After conversion, the new virtual CPU and motherboard you see won’t look very much like the old ones. CPU extensions like SSE, vectors, NX might come or go. The apparent CPU model and manufacturer might have changed. Therefore if you spent any time optimizing the installed packages, those optimizations are wasted and might even cause programs to crash.

It’s better to make your packages generic and have programs detect the runtime environment and optimizations needed each time they start up. (Actually, it’s worse than this when you also consider live migration, because the processor might change while a program is running — no one has really solved this one satisfactorily yet).

---

As you can see, P2V and V2V conversions are like major hardware changes. Your distro should be able to handle just about all the hardware changing underneath it. It should probe everything at boot, hard-code nothing, and be designed to cope gracefully with abrupt changes.

Update: A thread on fedora-devel-list about this issue.