http://people.redhat.com/~rjones/libguestfs-RHEL-6.3-preview/

These are based on libguestfs 1.16.1.

Here is an example program (fixed and updated):

const Guestfs = imports.gi.Guestfs;

function inspect (filename)
{
    var g = new Guestfs.Session ();

    //g.set_trace (true);

    var optargs = new Guestfs.AddDriveOpts ({readonly: true});
    g.add_drive_opts (filename, optargs);

    g.launch ();

    var roots = g.inspect_os ()
    if (roots.length == 0)
        printerr ("inspection: no operating systems found in", filename);
    else {
        for (var i = 0; i < roots.length; ++i) {
            inspect_root (g, roots[i]);
        }
    }
}

function inspect_root (g, root)
{
    print ("inspecting operating system root", root);

    print ("  product name:", g.inspect_get_product_name (root));
    print ("  version:",
           g.inspect_get_major_version (root),
           g.inspect_get_minor_version (root));
    //print ("  type:", g.inspect_get_type (root));
    print ("  distro:", g.inspect_get_distro (root));

    // Mount up the disks like guestfish -i
    var mps = g.inspect_get_mountpoints (root);
    var keys = [];
    for (var key in mps) { keys.push (key); }
    function compare (a, b) {
        if (a.length > b.length) return 1;
        else if (a.length == b.length) return 0;
        else return -1;
    }
    keys.sort (compare);

    for (var i = 0; i < keys.length; ++i) {
        g.mount_ro (mps[keys[i]], keys[i]);
    }

    // Get the list of applications.
    print ("  applications:");
    apps = g.inspect_list_applications (root);
    for (var i = 0; i < apps.length; ++i) {
        print ("    ", apps[i].app_name,
               apps[i].app_version, apps[i].app_release);
    }

    g.umount_all ();
}

if (ARGV.length != 1) {
    printerr ("Usage: gjs test.js disk.img");
} else {
    inspect (ARGV[0]);
}

One highlight is GObject bindings, which makes the API available from Javascript. More on this topic coming soon.

Why is there this difference? It comes down to how live migration is implemented.

Live migration is completely different from shutting down and copying a guest. During live migration we must send the complete state of system RAM, virtual CPUs, and all virtual devices, over to the remote side. In qemu this is done by sending “VMState” structures over the wire, one struct for each device that the guest is using. These structures are mostly a memory dump, but so that you don’t need byte-for-byte compatible versions of qemu when live migrating, each struct is preceded by a version ID.

The receiving qemu checks that it can handle that version of the struct. In some (but not all) cases, qemu knows how to “upgrade”, say, a version 1 struct into a version 2 struct. Downgrades are never possible, and some upgrades are also rejected (eg. if version 2 is a complete rewrite over version 1, then it’s possible for a device to refuse to deal with version 1 structs at all).

Downgrades are not possible, and that’s the basic reason why live migration doesn’t work from a newer to an older version of qemu.

Why does copying work? When a VM is shut down, there is no RAM, vCPU or device state. All the state that remains is the contents of the hard disk. If the hard disk is booted on an older qemu, then the kernel, during boot, will test the available CPUs, devices, etc and adjust itself, exactly the same as if you took a physical hard disk and transplanted it between real machines.

Indirectly related to all this is the qemu machine type. If you created guests on RHEL 6.0, then you may notice the libvirt XML contains:

<type arch='x86_64' machine='rhel6.0.0'>hvm</type>

This machine type stays with the guest even when you update the host.

The machine type controls what devices and PCI slots we present to the guest at boot, and it’s mainly there so that Windows doesn’t try to reactivate itself when you upgrade your host. The newer qemu presents the old devices and PCI assignments, so Windows doesn’t “notice” the updated hypervisor.

For Linux guests this is usually not a problem you have to worry about and you can go ahead and change the machine type at will.

Write something to a partition device (eg. /dev/vda1) and immediately call blockdev --rereadpt /dev/vda to re-read the partition table of the whole device. Sometimes (about 50% for me) the blockdev command fails with:

blockdev: BLKRRPART: Device or resource busy

Nothing else is using /dev/vda, nothing from it was mounted, and the error was intermittent which indicates a race condition.

Why this happens:

udev has a rule that runs blkid -o udev -p /dev/vda1. It does this every time you close a block device so that blkid can rescan the content of the device.

The act of blkid running very briefly behind our backs causes the device to be open during the blockdev operation, causing it to fail.

Adding udevadm settle between the close and the blockdev fixed the problem for us, although this command is also inherently racy (what happens if it runs before the kernel has sent a message to udev?)

Usage is quite simple:

$ virt-format -a disk.img

(Note that erases any data on disk.img!)

This works for any format of disk, eg. qcow2, or you can run it on host partitions, LVs, USB storage etc. By default it just creates a partition, but using other options you can make empty filesystems and logical volumes.

This is a simple tool that doesn’t cover everything you might want to do. For more complex requirements, see virt-make-fs or guestfish.

The name is a play on “wrapper” and “API”. And a play on what has been discussed for a very long time inside Red Hat — an API for everything that Red Hat Enterprise Linux does. A “rh-api”, if you will.

Wrappi aims to wrap up everything you can do with a Linux box:

• many POSIX functions for creating files, mkdir, reading and writing files, etc
• many command line tools, mkfs, fsck, parted, pvcreate, cp etc
• block devices, firewall rules, services, processes etc
• reboot, shutdown
• a way to read and modify all configuration files
• access to all the performance stats

We take all of that and present it back to you as an API:

• available from C and a dozen other programming languages
• available in a more powerful shell than bash, something like Microsoft’s PowerShell
• accessible remotely over ssh, HTTP, XML-RPC, dbus, and a dozen other protocols

The aim is that from a PowerShell-like interface (or your own program) you could control and maintain 100s of Linux machines remotely. Like puppet/chef, but at a lower level.

Here is what a C program using the API might look like.

So this is ambitious.

An API that did all of the above might run to 10,000 different calls. To make this feasible to maintain and implement, we have to be able to generate the code for just about everything.

Each API call starts with a description. Because we need to write this description for 10,000 calls, it’s best to keep this short. In the best case, just a single line would be required, but in some cases it’ll be more complicated:

entry_point void mkdir (pathname path, fileperm perm)

system_object block_device
  dir_list "/sys/block/[hsv]d[a-z]*"

entry_point
struct timeval gettimeofday ()
<<
  int r;
  struct timeval tv;

  r = gettimeofday (&tv, NULL);
  if (r == -1) {
    set_error_errno ("gettimeofday");
    return NULL;
  }
  ret->tv_sec = tv.tv_sec;
  ret->tv_usec = tv.tv_usec;
  return ret;
>>
includes ["sys/time.h"]

From that metadata we can generate automatically everything we need, all the language bindings, all the remote access code, the implementation.

When a new shiny-thing comes along (“JSON-powered Enterprise GObject”) we can simply add a new generator backend, and we’ll support the whole API through that.

If you’ve followed my blog, you’ll know that this looks a lot like the libguestfs generator, and in many ways this is the generator done right.

There’s lots of code in the git repo. If you want to find out more, follow the blog, or ask questions below!

The very unfortunate part about this is it easily allows guests to read and write parts of host devices that they are not supposed to. For example, if a guest was confined to host device /dev/sda3, it could read or write other partitions or the boot sector on /dev/sda.

In your guest, try this command which reads the host boot sector:

sg_dd if=/dev/vda blk_sgio=1 bs=512 count=1 of=output

Swap the if and of arguments around to exploit the host.

Here’s Paolo’s write-up on LKML.

Here is the libguestfs mitigation patch. The libvirt mitigation patch.

Like this crap because we did “gettextize” without doing “autoconf” (or vice versa):

make[2]: Entering directory `/builddir/build/BUILD/libguestfs-1.14.7/po'
*** error: gettext infrastructure mismatch: using a Makefile.in.in from gettext version 0.17 but the autoconf macros are from gettext version 0.18

I just don’t get why gettext can’t be a regular, ordinary, plain library so we don’t have to constantly suffer from this sort of thing. There is surely no other library that needs to rewrite your entire build system.

Exhibit (a): My Google Nexus S phone: requires rebooting every 3 days otherwise it just stops being able to receive/send calls and/or texts. Data over 3g still works during these outages.

Exhibit (b): My new Samsung Galaxy tab, a Christmas present of sorts from Red Hat: Fresh out of the box(!), this cannot connect to my wifi (absolutely everything else can connect fine to this wifi). Support forums are full of people reporting the same thing, and everyone suggesting you reboot/reinstall/press some magic key sequence/etc. Where have I heard that story before?

https://www.redhat.com/archives/libguestfs/2011-December/msg00061.html

><fs> blkid /dev/vg_f15x32/lv_root
UUID: 9293385f-3200-4694-8f4b-e20bb8d73c37
VERSION: 1.0
TYPE: ext4
USAGE: filesystem
MINIMUM_IO_SIZE: 512
PHYSICAL_SECTOR_SIZE: 512
LOGICAL_SECTOR_SIZE: 512
><fs> blkid /dev/vda1
UUID: d2cd4319-f515-4be2-9a5c-fc8b57b53723
VERSION: 1.0
TYPE: ext4
USAGE: filesystem
MINIMUM_IO_SIZE: 512
PHYSICAL_SECTOR_SIZE: 512
LOGICAL_SECTOR_SIZE: 512
PART_ENTRY_SCHEME: dos
PART_ENTRY_TYPE: 0x83
PART_ENTRY_FLAGS: 0x80
PART_ENTRY_NUMBER: 1
PART_ENTRY_OFFSET: 2048
PART_ENTRY_SIZE: 1024000
PART_ENTRY_DISK: 252:0

Thanks to Wanlong Gao [awesome photo!] for adding this.

Kevin, you are calling group 3 a “vocal minority”. But you present no hard data about how many people are actually in each group. So there is no evidence on which to say that this group is a minority, or even that the responses break down into these four groups at all. You hope that group 2 is large, but where’s the evidence that people really did check their firewalls and re-enable SELinux as a result of this pointless key change?

http://libguestfs.org/download/seminars/2011-11-18-introduction-to-libguestfs/

How did I do it in the end? Using ffmpeg on the command line and the editing script from my previous posting.

#!/bin/bash -

set -e

input="libguestfs-20111118 1602-1.mp4"
output="libguestfs-webinar-2011-11-18-final.avi"

codec="-b 500k -vcodec libx264 -ab 128k"
codec_novideo="-vn -ab 128k"
codec_noaudio="-b 500k -vcodec libx264"

cd /tmp

# part1: verbatim
# part2: replace video with "network failure" slide
# part3: verbatim
# part4: replace video with slide #5 from handout
# part5: verbatim
# part6: replace video with slide #1 from handout

split () {
    rm -f part1.avi part2_audio.avi part3.avi part4_audio.avi part5.avi part6_audio.avi

    ffmpeg -i "$input" -t 00:11:30 $codec part1.avi

    # end 13:33
    ffmpeg -i "$input" -ss 00:11:30 -t 00:02:03 $codec_novideo part2_audio.avi

    # end 32:03
    ffmpeg -i "$input" -ss 00:20:09 -t 00:11:54 $codec part3.avi

    # end 34:20
    ffmpeg -i "$input" -ss 00:32:03 -t 00:02:17 $codec_novideo part4_audio.avi

    # end 40:46
    ffmpeg -i "$input" -ss 00:34:20 -t 00:06:26 $codec part5.avi

    # end 49:32
    ffmpeg -i "$input" -ss 00:41:55 -t 00:07:37 $codec_novideo part6_audio.avi
}

static_slides () {
    # to replace part2 video
    rm -f part2_video.avi
    ffmpeg -loop_input -i /tmp/network_failure.png -t 00:02:03 \
        -vf pad='1024:768' -r 8 $codec_noaudio part2_video.avi

    # to replace part4 video
    rm -f part4_video.avi
    ffmpeg -loop_input -i /tmp/slide6.png -t 00:02:17 \
        -vf pad='1024:768:iw/2' -r 8 $codec_noaudio part4_video.avi

    # to replace part6 video
    rm -f part6_video.avi
    ffmpeg -loop_input -i /tmp/slide1.png -t 00:07:37 \
        -vf pad='1024:768:iw/2' -r 8 $codec_noaudio part6_video.avi
}

combine () {
    # combine audio and video
    rm -f part2.avi
    ffmpeg -i part2_audio.avi -i part2_video.avi $codec part2.avi
    rm -f part4.avi
    ffmpeg -i part4_audio.avi -i part4_video.avi $codec part4.avi
    rm -f part6.avi
    ffmpeg -i part6_audio.avi -i part6_video.avi $codec part6.avi
}

assemble () {
    rm -f "$output"
    # Concatenate parts into final video.  Note no transcoding here.
    mencoder part{1,2,3,4,5,6}.avi -ovc copy -oac copy -o "$output"
}

split
static_slides
combine
assemble

The final video isn’t a fully open format, in that it uses patented (in the US) H.264 and MP3. However it is fully playable with free software, particularly outside the US. The other advantage of using libx264 is it resulted in the smallest file size of all the codecs I tried.

Here is the unedited video:

http://oirase.annexia.org/tmp/libguestfs-20111118%201602-1.mp4 (13758086 bytes)

And here are my video editing instructions!

Note the video is straight from webex, untouched, so it’s in a proprietary format (avc1 video, mp4a audio). Most Linux players will play it just fine.

kdenlive. Usability FAIL. You can’t just load a video clip and edit it, unless it fits a “profile”. This video clip is a screen capture at 1024×768 so of course it doesn’t fit any existing profile like HD, DVD, 720p etc.

Why is video editing so hard?