Tag Archives: virtualization

Inspection, now with added Prolog

You can give libguestfs an unknown disk image, and it tries to guess what’s on it, in terms of operating systems, Linux distro, Windows drive letter mappings and so on, a process that we call inspection. This is an important part of many of the virt tools, because when you type a command like

$ virt-cat -a linux.img /var/log/messages

how is virt-cat to know that on this particular disk image the sysadmin put /var on a separate partition? Because, inspection.

Given that inspection is such an important part of many tools, and vital for standalone programs like virt-inspector you might wonder how it works.

The answer, right now, is 6000+ lines of hairy, intricate C code, which is difficult to maintain and a source of hard to fix bugs and hard to implement feature requests.

$ wc -l src/inspect*.c
   823 src/inspect-apps.c
   725 src/inspect.c
   777 src/inspect-fs.c
   543 src/inspect-fs-cd.c
  2092 src/inspect-fs-unix.c
   704 src/inspect-fs-windows.c
   600 src/inspect-icon.c
  6264 total

How can we make this better?

Getting back to basics, inspection is really a lot of heuristics. Things like:

  • If this filesystem contains a file /etc/fstab then it could be a Linux root filesystem. And:
  • If this thing we think is a Linux root filesystem contains /etc/debian_version then it could be a Debian root filesystem.

These heuristics can be expressed in a logic language inspired by Prolog:

LinuxRoot(fs) :-
    Filesystem(fs),
    File(fs, "/etc/fstab").
DebianRoot(fs) :-
    LinuxRoot(fs),
    File(fs, "/etc/debian_version").

What we’re doing here is collecting a set of facts (Prolog calls them “compound terms”), like:

Filesystem("/dev/sda1").
File("/dev/sda1", "/etc/fstab").
File("/dev/sda1", "/etc/debian_version").

and deriving new facts using the rules:

LinuxRoot("/dev/sda1").
DebianRoot("/dev/sda1").

(I should say at this point that I’m simplifying things a bit. If you want to get a flavour of what the inspection rules might finally look like, then take a look at this file.)

So far I have written a compiler that compiles inspection rules into fairly efficient C code (and hence to binaries), using a forward chaining strategy. It has some nice features like transparently embedding C code into the rules, allowing you to do more complicated operations directly in C:

Distro(fs, distro) :-
    LinuxRootWithOSRelease(fs), /* has /etc/os-release */
    (distro)?={{
      int r;
      CLEANUP_FREE char *distro = NULL;
      if ((r = get_distro_from_os_release (fs, &distro))
           <= 0)
        return r;
      set_distro (distro);
      return 0;
    }}.

and

BlockDevice(dev) :-
    (dev)*={{
      CLEANUP_FREE_STRING_LIST char **devs =
        get_all_block_devices ();
      if (devs == NULL) return -1;
      for (size_t i = 0; devs[i] != NULL; ++i)
        set_dev (devs[i]);
      return 0;
    }}.

My inspection rules run to < 500 lines of code so far, although it’s hard to compare that to the current code because (a) the inspection rules will likely double or triple in size once they are able to do everything that the current code can do, and (b) there’s a lot of supporting runtime code like get_all_block_devices above.

Nevertheless I hope the new rules system will be faster, more supportable and extensible, and easier to understand than the current code. It will also be 100% backwards compatible with existing libguestfs users (since we never break compatibility).

You can follow development in this branch on github.

Update: Hacker News discussion of this article.

Advertisements

Leave a comment

Filed under Uncategorized

Linux Kernel Library backend for libguestfs

For a very long time I’ve been keeping an eye on rump kernels. These are a fancy way of saying ordinary kernel code which has been ported or altered in some way so you can link it to userspace programs, usually as a library. There have been claims that rump kernels make libguestfs obsolete, which is a load of nonsense for reasons I’ll talk about in a minute. But rump kernels are interesting and finally I’ve been able to add a rump kernel library to libguestfs.

The first time I encountered rump kernels was in 2010 with Prof. Renzo Davoli’s virtualsquare View-OS project. The more recent rump kernel project was based on NetBSD. Since NetBSD doesn’t use Linux drivers, but reimplementations, it was never very interesting. We couldn’t have included it in RHEL, and it would be only a matter of time before someone found an important feature or data-corrupting difference between the real ext2/3 driver from Linux and the reimplementation in BSD. Last I looked, NetBSD didn’t even support ext4. The choice of NetBSD made it a non-starter for us.

libos arrived earlier this year and, being based on Linux, was a lot more interesting, but their aim was only the Linux network stack so it wasn’t directly relevant to libguestfs, which is only concerned with disk and filesystem access.

However last week Richard Weinberger (Linux UML maintainer) CC’d me on a much more interesting project. LKL (Linux Kernel Library) is a relatively small patchset on top of Linux that turns Linux into a library. Quite literally you just link your program with -llkl and you can start a Linux kernel and make Linux system calls. A few example programs can be found here.

Today I wrote a work-in-progress LKL backend for libguestfs. [Note for those keen to jump in and use it, it doesn’t do very much at the moment. There is a small to-do list. But it’s only a few hours more work to have it as a useful backend for libguestfs.]

What could an LKL-enabled libguestfs do? Well it could open a raw disk image that contains only a Linux filesystem, or possibly a partitioned disk image with a Linux filesystem. And it can do that very quickly – in about 0.03 seconds in my tests (compared to 2.5 seconds for the qemu-based libguestfs).

There are a long list of things that LKL cannot do however. It cannot open anything containing LVM (because LVM requires userspace tools, that would all have to be modified to use liblkl, and there would be all kinds of synchronization problems between the LVM-linked liblkl kernel and the libguestfs-linked kernel). It cannot run anything that requires a userspace tool. That includes btrfs, ntfs-3g (FUSE based), some MD devices, encrypted disks, dm-thinp and Windows Dynamic Disks.

It cannot create new LVs or filesystems or any of the other disk structures that libguestfs can create.

It cannot open disk formats like qcow2 and vmdk. The qcow2 code in particular is very specific to qemu, and cannot possibly be ported to the kernel.

It cannot open remote devices like Ceph, Gluster, https, ssh or ftp (although there is an nbd client in the kernel, so one day that may be possible).

It cannot run commands in the context of the guest.

It’s also less secure than libguestfs with the qemu backend, because it doesn’t isolate your host from exploits in the guest or filesystem using virtualization and sVirt.

All of those things can be done by libguestfs today, but may never be possible (or only in limited cases) with LKL.

But within the limits of what LKL can do, it should become an excellent choice (assuming it gets upstream). Adding an LKL backend to libguestfs brings to the table the large, but clean and well-tested libguestfs API and the language bindings and virt tools, so libguestfs gives you the best of both worlds: the performance of LKL by using that backend, or the massive feature set of libguestfs by switching over to the qemu or libvirt backend, which for most programs is a single line code change.

13 Comments

Filed under Uncategorized

Tip: guestmount (FUSE mount) every filesystem in a disk image

Maxim asks an interesting question which is if you’ve got a disk image, how do you mount every filesystem onto your host. Like this:

$ ./fs-mount.pl rhel-5.11.img /tmp/fs &
$ cd /tmp/fs
/tmp/fs$ ls
dev
/tmp/fs$ cd dev
/tmp/fs/dev$ ls
sda1  sda2  sda3
/tmp/fs/dev$ cd sda2
/tmp/fs/dev/sda2$ ls
bin   dev  home  lib64       media  mnt  proc  sbin     srv  tmp  var
boot  etc  lib   lost+found  misc   opt  root  selinux  sys  usr
...
$ cd /tmp
$ guestunmount /tmp/fs

The answer is this surprisingly short Perl script.

#!/usr/bin/perl

use warnings;
use strict;

use Sys::Guestfs;

die "usage: $0 disk1 [disk2 ...] mountpoint\n" if @ARGV <= 1;

my $mp = pop;

my $g = Sys::Guestfs->new ();
foreach (@ARGV) {
    $g->add_drive ($_);
}
$g->launch ();

# Examine the filesystems.
my %fses = $g->list_filesystems ();

# Create the mountpoint directories (in the libguestfs namespace)
# and mount the filesystems on them.
foreach my $fs (sort keys %fses) {
    # mkmountpoint is really the same as mkdir.  Unfortunately there
    # is no 'mkdir -p' equivalent, so we have to do this instead:
    my @components = split ("/", $fs);
    for (my $i = 1; $i < @components; ++$i) {
        my $dir = "/" . join ("/", @components[1 .. $i]);
        eval { $g->mkmountpoint ($dir) }
    }

    # Don't fail if the filesystem can't be mounted, eg. it's swap.
    eval { $g->mount ($fs, $fs) }
}

# Export the filesystem on the host.
$g->mount_local ($mp);
$g->mount_local_run ();

# Close nicely since we mounted everything writable.
$g->shutdown ();
$g->close ();

Leave a comment

Filed under Uncategorized

Virt-builder Fedora 23 image

Fedora 23 was released today. Get it through virt-builder in just two simple commands:

$ virt-builder fedora-23 \
    --root-password password:123456 \
    --size 20G
$ qemu-system-x86_64 -drive file=fedora-23,if=virtio \
    -m 2048

f23

2 Comments

Filed under Uncategorized

virt-v2v, libguestfs and qemu remote drivers in RHEL 7

Upstream qemu can access a variety of remote disks, like NBD and Ceph. This feature is exposed in libguestfs so you can easily mount remote storage.

However in RHEL 7 many of these drivers are disabled, because they’re not stable enough to support. I was asked exactly how this works, and this post is my answer — as it’s not as simple as it sounds.

There are (at least) five separate layers involved:

qemu code What block drivers are compiled into qemu, and which ones are compiled out completely.
qemu block driver r/o whitelist A whitelist of drivers that qemu allows you to use read-only.
qemu block driver r/w whitelist A whitelist of drivers that qemu allows you to use for read and write.
libvirt What libvirt enables (not covered in this discussion).
libguestfs In RHEL we patch out some qemu remote storage types using a custom patch.

Starting at the bottom of the stack, in RHEL we use ./configure --disable-* flags to disable a few features: Ceph is disabled on !x86_64 and 9pfs is disabled everywhere. This means the qemu binary won’t even contain code for those features.

If you run qemu-img --help in RHEL 7, you’ll see the drivers which are compiled into the binary:

$ rpm -qf /usr/bin/qemu-img
qemu-img-1.5.3-92.el7.x86_64
$ qemu-img --help
[...]
Supported formats: vvfat vpc vmdk vhdx vdi ssh
sheepdog rbd raw host_cdrom host_floppy host_device
file qed qcow2 qcow parallels nbd iscsi gluster dmg
tftp ftps ftp https http cloop bochs blkverify blkdebug

Although you can use all of those in qemu-img, not all of those drivers work in qemu (the hypervisor). qemu implements two whitelists. The RHEL 7 qemu-kvm.spec file looks like this:

./configure [...]
    --block-drv-rw-whitelist=qcow2,raw,file,host_device,blkdebug,nbd,iscsi,gluster,rbd \
    --block-drv-ro-whitelist=vmdk,vhdx,vpc,ssh,https

The --block-drv-rw-whitelist parameter configures the drivers for which full read and write access is permitted and supported in RHEL 7. It’s quite a short list!

Even shorter is the --block-drv-ro-whitelist parameter — drivers for which only read-only access is allowed. You can’t use qemu to open these files for write. You can use these as (read-only) backing files, but you can’t commit to those backing files.

In practice what happens is you get an error if you try to use non-whitelisted block drivers:

$ /usr/libexec/qemu-kvm -drive file=test.vpc
qemu-kvm: -drive file=test.vpc: could not open disk image
test.vpc: Driver 'vpc' can only be used for read-only devices
$ /usr/libexec/qemu-kvm -drive file=test.qcow1
qemu-kvm: -drive file=test.qcow1: could not open disk
image test.qcow1: Driver 'qcow' is not whitelisted

Note that’s a qcow v1 (ancient format) file, not modern qcow2.

Side note: Only qemu (the hypervisor) enforces the whitelist. Tools like qemu-img ignore it.

At the top of the stack, libguestfs has a patch which removes support for many remote protocols. Currently (RHEL 7.2/7.3) we disable: ftp, ftps, http, https, tftp, gluster, iscsi, sheepdog, ssh. That leaves only: local file, rbd (Ceph) and NBD enabled.

virt-v2v uses a mixture of libguestfs and qemu-img to convert VMware and Xen guests to run on KVM. To access VMware we need to use https and to access Xen we use ssh. Both of those drivers are disabled in libguestfs, and only available read-only in the qemu whitelist. However that’s sufficient for virt-v2v, since all it does is add the https or ssh driver as a read-only backing file. (If you are interested in finding out more about how virt-v2v works, then I gave a talk about it at the KVM Forum which is available online).

In summary — it’s complicated.

5 Comments

Filed under Uncategorized

Importing KVM guests to oVirt or RHEV

One of the tools I maintain is virt-v2v. It’s a program to import guests from foreign hypervisors like VMware and Xen, to KVM. It only does conversions to KVM, not the other way. And a feature I intentionally removed in RHEL 7 was importing KVM → KVM.

Why would you want to “import” KVM → KVM? Well, no reason actually. In fact it’s one of those really bad ideas for V2V. However it used to have a useful purpose: oVirt/RHEV can’t import a plain disk image, but virt-v2v knows how to import things to oVirt, so people used virt-v2v as backdoor for this missing feature.

Removing this virt-v2v feature has caused a lot of moaning, but I’m adamant it’s a very bad idea to use virt-v2v as a way to import disk images. Virt-v2v does all sorts of complex filesystem and Windows Registry manipulations, which you don’t want and don’t need if your guest already runs on KVM. Worst case, you could even end up breaking your guest.

However I have now written a replacement script that does the job: http://git.annexia.org/?p=import-to-ovirt.git

If your guest is a disk image that already runs on KVM, then you can use this script to import the guest. You’ll need to clone the git repo, read the README file, and then read the tool’s man page. It’s pretty straightforward.

There are a few shortcomings with this script to be aware of:

  1. The guest must have virtio drivers installed already, and must be able to boot off virtio-blk (default) or virtio-scsi. For virtio-scsi, you’ll need to flip the checkbox in the ‘Advanced’ section of the guest parameters in the oVirt UI.
  2. It should be possible to import guests that don’t have virtio drivers installed, but can use IDE. This is a missing feature (patches welcome).
  3. No network card is added to the guest, so it probably won’t have network when it boots. It should be possible to add a network card through the UI, but really this is something that needs to be fixed in the script (patches welcome).
  4. It doesn’t handle all the random packaging formats that guests come in, like OVA. You’ll have to extract these first and import just the disk image.
  5. It’s not in any way supported or endorsed by Red Hat.

21 Comments

Filed under Uncategorized

How to rebuild libguestfs from source on RHEL or CentOS 7

Three people have asked me about this, so here goes. You will need a RHEL or CentOS 7.1 machine (perhaps a VM), and you may need to grab extra packages from this preview repository. The preview repo will go away when we release 7.2, but then again 7.2 should contain all the packages you need.

You’ll need to install rpm-build. You could also install mock (from EPEL), but in fact you don’t need mock to build libguestfs and it may be easier and faster without.

Please don’t build libguestfs as root. It’s not necessary to build (any) packages as root, and can even be dangerous.

Grab the source RPM. The latest at time of writing is libguestfs-1.28.1-1.55.el7.src.rpm. When 7.2 comes out, you’ll be able to get the source RPM using this command:

yumdownloader --source libguestfs

I find it helpful to build RPMs in my home directory, and also to disable the libguestfs tests. To do that, I have a ~/.rpmmacros file that contains:

%_topdir	%(echo $HOME)/rpmbuild
%_smp_mflags	-j5
%libguestfs_runtests   0

You may wish to adjust %_smp_mflags. A good value to choose is 1 + the number of cores on your machine.

I’ll assume at this point that the reason you want to rebuild libguestfs is to apply a patch (otherwise why aren’t you using the binaries we supply?), so first let’s unpack the source tree. Note I am running this command as non-root:

rpm -i libguestfs-1.28.1-1.55.el7.src.rpm

If you set up ~/.rpmmacros as above then the sources should be unpacked under ~/rpmbuild/SPECS and ~/rpmbuild/SOURCES.

Take a look at least at the libguestfs.spec file. You may wish to modify it now to add any patches you need (add the patch files to the SOURCES/ subdirectory). You might also want to modify the Release: tag so that your package doesn’t conflict with the official package.

You might also need to install build dependencies. This command should be run as root since it needs to install packages, and also note that you may need packages from the repo linked above.

yum-builddep libguestfs.spec

Now you can rebuild libguestfs (non-root!):

rpmbuild -ba libguestfs.spec

With the tests disabled, on decent hardware, that should take about 10 minutes.

The final binary packages will end up in ~/rpmbuild/RPMS/ and can be installed as normal:

yum localupdate x86_64/*.rpm noarch/*.rpm

You might see errors during the build phase. If they aren’t fatal, you can ignore them, but if the build fails then post the complete log to our mailing list (you don’t need to subscribe) so we can help you out.

7 Comments

Filed under Uncategorized