Red Hat speakers at FOSDEM 2012

In just over 1 week, FOSDEM 2012 is being held in Brussels. This flyer (PDF) is a handy guide to the Red Hat folk who will be giving talks there. (Thanks Máirín Duffy).

libguestfs RHEL 6.3 new preview packages available

Here:

http://people.redhat.com/~rjones/libguestfs-RHEL-6.3-preview/

These are based on libguestfs 1.16.1.

Calling libguestfs from Javascript

In libguestfs 1.16 we added experimental GObject bindings and support for GObject Introspection. These are experimental because we may change them a little in future. They do allow you to access libguestfs from Javascript, specifically from gjs.

Here is an example program (fixed and updated):

const Guestfs = imports.gi.Guestfs;

function inspect (filename)
{
    var g = new Guestfs.Session ();

    //g.set_trace (true);

    var optargs = new Guestfs.AddDriveOpts ({readonly: true});
    g.add_drive_opts (filename, optargs);

    g.launch ();

    var roots = g.inspect_os ()
    if (roots.length == 0)
        printerr ("inspection: no operating systems found in", filename);
    else {
        for (var i = 0; i < roots.length; ++i) {
            inspect_root (g, roots[i]);
        }
    }
}

function inspect_root (g, root)
{
    print ("inspecting operating system root", root);

    print ("  product name:", g.inspect_get_product_name (root));
    print ("  version:",
           g.inspect_get_major_version (root),
           g.inspect_get_minor_version (root));
    //print ("  type:", g.inspect_get_type (root));
    print ("  distro:", g.inspect_get_distro (root));

    // Mount up the disks like guestfish -i
    var mps = g.inspect_get_mountpoints (root);
    var keys = [];
    for (var key in mps) { keys.push (key); }
    function compare (a, b) {
        if (a.length > b.length) return 1;
        else if (a.length == b.length) return 0;
        else return -1;
    }
    keys.sort (compare);

    for (var i = 0; i < keys.length; ++i) {
        g.mount_ro (mps[keys[i]], keys[i]);
    }

    // Get the list of applications.
    print ("  applications:");
    apps = g.inspect_list_applications (root);
    for (var i = 0; i < apps.length; ++i) {
        print ("    ", apps[i].app_name,
               apps[i].app_version, apps[i].app_release);
    }

    g.umount_all ();
}

if (ARGV.length != 1) {
    printerr ("Usage: gjs test.js disk.img");
} else {
    inspect (ARGV[0]);
}

libguestfs 1.16 has been released

The release notes are here.

One highlight is GObject bindings, which makes the API available from Javascript. More on this topic coming soon.

Why can’t you live migrate from newer to older versions of qemu/KVM?

I answered a question on a mailing list about live migration versus copying guests between different versions of KVM on RHEL. The complainant observed that you can’t live migrate from RHEL 6.2 to RHEL 6.1. But you can shut down a guest, copy it from RHEL 6.2 to 6.1, and boot it.

Why is there this difference? It comes down to how live migration is implemented.

Live migration is completely different from shutting down and copying a guest. During live migration we must send the complete state of system RAM, virtual CPUs, and all virtual devices, over to the remote side. In qemu this is done by sending “VMState” structures over the wire, one struct for each device that the guest is using. These structures are mostly a memory dump, but so that you don’t need byte-for-byte compatible versions of qemu when live migrating, each struct is preceded by a version ID.

The receiving qemu checks that it can handle that version of the struct. In some (but not all) cases, qemu knows how to “upgrade”, say, a version 1 struct into a version 2 struct. Downgrades are never possible, and some upgrades are also rejected (eg. if version 2 is a complete rewrite over version 1, then it’s possible for a device to refuse to deal with version 1 structs at all).

Downgrades are not possible, and that’s the basic reason why live migration doesn’t work from a newer to an older version of qemu.

Why does copying work? When a VM is shut down, there is no RAM, vCPU or device state. All the state that remains is the contents of the hard disk. If the hard disk is booted on an older qemu, then the kernel, during boot, will test the available CPUs, devices, etc and adjust itself, exactly the same as if you took a physical hard disk and transplanted it between real machines.

Indirectly related to all this is the qemu machine type. If you created guests on RHEL 6.0, then you may notice the libvirt XML contains:

<type arch='x86_64' machine='rhel6.0.0'>hvm</type>

This machine type stays with the guest even when you update the host.

The machine type controls what devices and PCI slots we present to the guest at boot, and it’s mainly there so that Windows doesn’t try to reactivate itself when you upgrade your host. The newer qemu presents the old devices and PCI assignments, so Windows doesn’t “notice” the updated hypervisor.

For Linux guests this is usually not a problem you have to worry about and you can go ahead and change the machine type at will.

udev unexpectedness

This was unexpected:

Write something to a partition device (eg. /dev/vda1) and immediately call blockdev --rereadpt /dev/vda to re-read the partition table of the whole device. Sometimes (about 50% for me) the blockdev command fails with:

blockdev: BLKRRPART: Device or resource busy

Nothing else is using /dev/vda, nothing from it was mounted, and the error was intermittent which indicates a race condition.

Why this happens:

udev has a rule that runs blkid -o udev -p /dev/vda1. It does this every time you close a block device so that blkid can rescan the content of the device.

The act of blkid running very briefly behind our backs causes the device to be open during the blockdev operation, causing it to fail.

Adding udevadm settle between the close and the blockdev fixed the problem for us, although this command is also inherently racy (what happens if it runs before the kernel has sent a message to udev?)

virt-format: Erase and make blank disks

There is a new tool in libguestfs ≥ 1.15.17: virt-format lets you erase disks and make blank disks.

Usage is quite simple:

$ virt-format -a disk.img

(Note that erases any data on disk.img!)

This works for any format of disk, eg. qcow2, or you can run it on host partitions, LVs, USB storage etc. By default it just creates a partition, but using other options you can make empty filesystems and logical volumes.

This is a simple tool that doesn’t cover everything you might want to do. For more complex requirements, see virt-make-fs or guestfish.

Details of the libguestfs talk at FOSDEM

… are here.

New project: wrappi: an API for Linux

Over the holidays I started a new project called “wrappi”.

The name is a play on “wrapper” and “API”. And a play on what has been discussed for a very long time inside Red Hat — an API for everything that Red Hat Enterprise Linux does. A “rh-api”, if you will.

Wrappi aims to wrap up everything you can do with a Linux box:

• many POSIX functions for creating files, mkdir, reading and writing files, etc
• many command line tools, mkfs, fsck, parted, pvcreate, cp etc
• block devices, firewall rules, services, processes etc
• reboot, shutdown
• a way to read and modify all configuration files
• access to all the performance stats

We take all of that and present it back to you as an API:

• available from C and a dozen other programming languages
• available in a more powerful shell than bash, something like Microsoft’s PowerShell
• accessible remotely over ssh, HTTP, XML-RPC, dbus, and a dozen other protocols

The aim is that from a PowerShell-like interface (or your own program) you could control and maintain 100s of Linux machines remotely. Like puppet/chef, but at a lower level.

Here is what a C program using the API might look like.

So this is ambitious.

An API that did all of the above might run to 10,000 different calls. To make this feasible to maintain and implement, we have to be able to generate the code for just about everything.

Each API call starts with a description. Because we need to write this description for 10,000 calls, it’s best to keep this short. In the best case, just a single line would be required, but in some cases it’ll be more complicated:

entry_point void mkdir (pathname path, fileperm perm)

system_object block_device
  dir_list "/sys/block/[hsv]d[a-z]*"

entry_point
struct timeval gettimeofday ()
<<
  int r;
  struct timeval tv;

  r = gettimeofday (&tv, NULL);
  if (r == -1) {
    set_error_errno ("gettimeofday");
    return NULL;
  }
  ret->tv_sec = tv.tv_sec;
  ret->tv_usec = tv.tv_usec;
  return ret;
>>
includes ["sys/time.h"]

From that metadata we can generate automatically everything we need, all the language bindings, all the remote access code, the implementation.

When a new shiny-thing comes along (“JSON-powered Enterprise GObject”) we can simply add a new generator backend, and we’ll support the whole API through that.

If you’ve followed my blog, you’ll know that this looks a lot like the libguestfs generator, and in many ways this is the generator done right.

There’s lots of code in the git repo. If you want to find out more, follow the blog, or ask questions below!

guestfish tutorial in Linux Format #152

It was out a couple of weeks ago. Previously …