Tip: Changing the qemu product name in libguestfs

20:30 < koike> Hi. Is it possible to configure the dmi codes for libguestfs? I mean, I am running cloud-init inside a libguestfs session (through python-guestfs) in GCE, the problem is that cloud-init reads /sys/class/dmi/id/product_name to determine if the machine is a GCE machine, but the value it read is Standard PC (i440FX + PIIX, 1996) instead of the expected Google Compute Engine so cloud-init fails.

The answer is yes, using the guestfs_config API that lets you set arbitrary qemu parameters:

g.config('-smbios',
         'type=1,product=Google Compute Engine')

Tip: Run virt-inspector on a compressed disk (with nbdkit)

virt-inspector is a very convenient tool to examine a disk image and find out if it contains an operating system, what applications are installed and so on.

If you have an xz-compressed disk image, you can run virt-inspector on it without uncompressing it, using the magic of captive nbdkit. Here’s how:

nbdkit xz file=win7.img.xz \
    -U - \
    --run 'virt-inspector --format=raw -a nbd://?socket=$unixsocket'

What’s happening here is we run nbdkit with the xz plugin, and tell it to serve NBD over a randomly named Unix domain socket (-U -).

We then run virt-inspector as a sub-process. This is called “captive nbdkit”. (Nbdkit is “captive” here, because it will exit as soon as virt-inspector exits, so there’s no need to clean anything up.)

The $unixsocket variable expands to the name of the randomly generated Unix domain socket, forming a libguestfs NBD URL which allows virt-inspector to examine the raw uncompressed data exported by nbdkit.

The nbdkit xz plugin only uncompresses those blocks of the data which are actually accessed, so this is quite efficient.

Tip: Poor man’s qemu breakpoint

I’ve written before about how you can use qemu + gdb to debug a guest. Today I was wondering how I was going to debug a problem in a BIOS option ROM, when Stefan Hajnoczi mentioned this tip: Insert

1: jmp 1b

into the code as a “poor man’s breakpoint”. In case you don’t know what that assembly code does, it causes a jump back (b) to the previous 1 label. In other words, an infinite loop.

After inserting that into the option ROM, recompiling and rebooting the virtual machine, it hangs in the boot, and hitting ^C in gdb gets me straight to the place where I inserted the loop.

(gdb) target remote localhost:1234
Remote debugging using localhost:1234
0x0000fff0 in ?? ()
(gdb) set architecture i8086
The target architecture is assumed to be i8086
(gdb) cont
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x00000045 in ?? ()
(gdb) info registers
eax            0xc100	49408
ecx            0x0	0
edx            0x0	0
ebx            0x0	0
esp            0x6f30	0x6f30
ebp            0x6f30	0x6f30
esi            0x0	0
edi            0x0	0
eip            0x45	0x45
eflags         0x2	[ ]
cs             0xc100	49408
ss             0x0	0
ds             0xc100	49408
es             0x0	0
fs             0x0	0
gs             0x0	0
(gdb) disassemble 0xc1000,0xc1050
Dump of assembler code from 0xc1000 to 0xc1050:
...
   0x000c103c:	mov    %cs,%ax
   0x000c103e:	mov    %ax,%ds
   0x000c1040:	mov    %esp,%ebp
   0x000c1043:	cli    
   0x000c1044:	cld    
   0x000c1045:	jmp    0xc1045
   0x000c1047:	jmp    0xc162c
   0x000c104a:	sub    $0x4,%esp
   0x000c104e:	mov    0xc(%esp),%eax
End of assembler dump.

Look, my infinite loop!

I can then jump over the loop and keep single stepping^*:

(gdb) set $eip=0x47
(gdb) si
0x0000062c in ?? ()
(gdb) si
0x0000062e in ?? ()
(gdb) si
0x00000632 in ?? ()

I did wonder if I could take Stefan’s idea further and insert an actual breakpoint (int $3) into the code, but that didn’t work for me.

Note to set breakpoints, the regular gdb break command doesn’t work. You have to use hardware-assisted breakpoints instead:

(gdb) hbreak *0xc164a
Hardware assisted breakpoint 1 at 0xc164a
(gdb) cont
Continuing.

Program received signal SIGTRAP, Trace/breakpoint trap.
0x0000064a in ?? ()

Note:
^* If you find that single stepping doesn’t work, make sure you are using qemu in TCG mode (-M accel=tcg), as KVM code apparently cannot be single-stepped.

Tip: FUSE-mount a disk image with Windows drive letters

guestmount is the libguestfs tool for taking a disk image and mounting it under the host filesystem. This works great for Linux disk images:

$ virt-builder centos-7.2
$ mkdir /tmp/mnt
$ guestmount -a centos-7.2.img -i /tmp/mnt
$ ls /tmp/mnt
bin   dev  home  lib64       media  opt   root  sbin  sys  usr
boot  etc  lib   lost+found  mnt    proc  run   srv   tmp  var
$ guestunmount /tmp/mnt

Those files under /tmp/mnt are inside the centos-7.2.img disk image file, and you can read and write them.

guestmount is fine for Windows disk images too, except when Windows has multiple drives, C:, D:, etc., because in that case you’ll only “see” the contents of the C: drive.

But guestmount is nowadays just a wrapper around the “mount-local” API in libguestfs, and you can use that API directly if you want to do anything a bit more complicated … such as exposing Windows drive letters.

Here is a Perl script which uses the mount-local API directly to do this:

#!/usr/bin/perl -w
use strict;
use Sys::Guestfs;
$| = 1;
die "usage: $0 mountpoint disk.img" if @ARGV < 2;
my $mp = shift @ARGV;
my $g = new Sys::Guestfs;
$g->add_drive_opts ($_) foreach @ARGV;
$g->launch;
my @roots = $g->inspect_os;
die "$0: no operating system found" if @roots != 1;
my $root = $roots[0];
die "$0: not Windows" if $g->inspect_get_type ($root) ne "windows";
my %map = $g->inspect_get_drive_mappings ($root);
foreach (keys %map) {
    $g->mkmountpoint ("/$_");
    eval { $g->mount ($map{$_}, "/$_") };
    warn "$@ (ignored)\n" if $@;
}
$g->mount_local ($mp);
print "filesystem mounted on $mp\n";
$g->mount_local_run;
$g->shutdown;

You can use it like this:

$ mkdir /tmp/mnt
$ ./drive-letters.pl /tmp/mnt windows7.img
filesystem ready on /tmp/mnt

in another window:

$ cd /tmp/mnt
$ ls
C  D
$ cd C
$ ls
Documents and Settings
PerfLogs
ProgramData
Program Files
[etc]
$ cd ../..
$ guestunmount /tmp/mnt

(Thanks to Pino Toscano for working out the details)

Tip: guestmount (FUSE mount) every filesystem in a disk image

Maxim asks an interesting question which is if you’ve got a disk image, how do you mount every filesystem onto your host. Like this:

$ ./fs-mount.pl rhel-5.11.img /tmp/fs &
$ cd /tmp/fs
/tmp/fs$ ls
dev
/tmp/fs$ cd dev
/tmp/fs/dev$ ls
sda1  sda2  sda3
/tmp/fs/dev$ cd sda2
/tmp/fs/dev/sda2$ ls
bin   dev  home  lib64       media  mnt  proc  sbin     srv  tmp  var
boot  etc  lib   lost+found  misc   opt  root  selinux  sys  usr
...
$ cd /tmp
$ guestunmount /tmp/fs

The answer is this surprisingly short Perl script.

#!/usr/bin/perl

use warnings;
use strict;

use Sys::Guestfs;

die "usage: $0 disk1 [disk2 ...] mountpoint\n" if @ARGV <= 1;

my $mp = pop;

my $g = Sys::Guestfs->new ();
foreach (@ARGV) {
    $g->add_drive ($_);
}
$g->launch ();

# Examine the filesystems.
my %fses = $g->list_filesystems ();

# Create the mountpoint directories (in the libguestfs namespace)
# and mount the filesystems on them.
foreach my $fs (sort keys %fses) {
    # mkmountpoint is really the same as mkdir.  Unfortunately there
    # is no 'mkdir -p' equivalent, so we have to do this instead:
    my @components = split ("/", $fs);
    for (my $i = 1; $i < @components; ++$i) {
        my $dir = "/" . join ("/", @components[1 .. $i]);
        eval { $g->mkmountpoint ($dir) }
    }

    # Don't fail if the filesystem can't be mounted, eg. it's swap.
    eval { $g->mount ($fs, $fs) }
}

# Export the filesystem on the host.
$g->mount_local ($mp);
$g->mount_local_run ();

# Close nicely since we mounted everything writable.
$g->shutdown ();
$g->close ();

Tip: Updating RHEL 7.1 cloud images using virt-customize and subscription-manager

Red Hat provide RHEL KVM guest and cloud images. At time of writing, the last one was built in Feb 2015, and so undoubtedly contains packages which are out of date or insecure.

You can use virt-customize to update the packages in the cloud image. This requires the libguestfs subscription-manager feature which will only be available in RHEL 7.3, but see here for RHEL 7.3 preview packages. Alternatively you can use Fedora ≥ 22.

$ virt-customize \
  -a rhel-guest-image-7.1-20150224.0.x86_64.qcow2 \
  --sm-credentials 'USERNAME:password:PASSWORD' \
  --sm-register --sm-attach auto \
  --update
[   0.0] Examining the guest ...
[  17.2] Setting a random seed
[  17.2] Registering with subscription-manager
[  28.8] Attaching to compatible subscriptions
[  61.3] Updating core packages
[ 976.8] Finishing off

1. You should probably use --sm-credentials USERNAME:file:FILENAME to specify your password using a file, rather than having it exposed on the command line.
2. The command above will leave the image template registered to RHN. To unregister it, add --sm-unregister at the end.

Tip: virt-install Windows with virtio device drivers

You have to unset these variables because of a long-standing bug in SPICE:

# unset http_proxy
# unset https_proxy

You can’t use virt-install’s --cdrom option twice, because virt-install ignores the second use of the option and only adds a single CD-ROM to the guest. Instead, use --disk ...,device=cdrom,bus=ide:

# virt-install --name=w81-virtio --ram=4096 \
    --cpu=host --vcpus=2 \
    --os-type=windows --os-variant=win8.1 \
    --disk /dev/VG/w81-virtio,bus=virtio \
    --disk en-gb_windows_8.1_pro_n_vl_with_update_x64_dvd_6050975.iso,device=cdrom,bus=ide \
    --disk /usr/share/virtio-win/virtio-win.iso,device=cdrom,bus=ide

During the install you’ll have to select the “Load driver” option and load the right viostor driver from the second CD-ROM (E:).

Tip: Wake up a guest from screen blank

A few years ago Dan Berrange added a way to send fake keyboard events to libvirt guests. You can use this to inject just a press on the Left Shift key to wake up a guest from screen blank. Very useful if you need to take a screenshot!

$ virsh send-key guest KEY_LEFTSHIFT
$ sleep 1
$ virsh screenshot guest /tmp/screenshot.ppm

Update: A word of warning though. If you try this for Windows guests you’ll hit this message:

The solution is to hit other keys randomly. Grrr.

Tip: Read guest disks from VMware vCenter using libguestfs

virt-v2v can import guests directly from vCenter. It uses all sorts of tricks to make this fast and efficient, but the basic technique uses plain https range requests.

Making it all work was not so easy and involved a lot of experimentation and bug fixing, and I don’t think it has been documented up to now. So this post describes how we do it. As usual the code is the ultimate repository of our knowledge so you may want to consult that after reading this introduction.

Note this is read-only access. Write access is possible, but you’ll have to use ssh instead.

VMware ESXi hypervisor has a web server but doesn’t support range requests, so although you can download an entire disk image in one go from the ESXi hypervisor, to random-access the image using libguestfs you will need VMware vCenter. You should check that virsh dumpxml works against your vCenter instance by following these instructions. If that doesn’t work, it’s unlikely the rest of the instructions will work.

You will need to know:

1. The hostname or IP address of your vCenter server,
2. the username and password for vCenter,
3. the name of your datacenter (probably Datacenter),
4. the name of the datastore containing your guest (could be datastore1),
5. .. and of course the name of your guest.

Tricky step 1 is to construct the vCenter https URL of your guest.

This looks like:

https://root:password@vcenter/folder/guest/guest-flat.vmdk?dcPath=Datacenter&dsName=datastore1

where:

root:password

username and password

vcenter

vCenter hostname or IP address

guest

guest name (repeated twice)

Datacenter

datacenter name

datastore1

datastore

Once you’ve got a URL that looks right, try to fetch the headers using curl. This step is important! not just because it checks the URL is good, but because it allows us to get a cookie which is required else vCenter will break under the load when we start to access it for real.

$ curl --insecure -I https://....
HTTP/1.1 200 OK
Date: Wed, 5 Nov 2014 19:38:32 GMT
Set-Cookie: vmware_soap_session="52a3a513-7fba-ef0e-5b36-c18d88d71b14"; Path=/; HttpOnly; Secure; 
Accept-Ranges: bytes
Connection: Keep-Alive
Content-Type: application/octet-stream
Content-Length: 8589934592

The cookie is the vmware_soap_session=... part including the quotes.

Now let’s make a qcow2 overlay which encodes our https URL and the cookie as the backing file. This requires a reasonably recent qemu, probably 2.1 or above.

$ qemu-img create -f qcow2 /tmp/overlay.qcow2 \
    -b 'json: { "file.driver":"https",
                "file.url":"https://..",
                "file.cookie":"vmware_soap_session=\"...\"",
                "file.sslverify":"off",
                "file.timeout":1000 }'

You don’t need to include the password in the URL here, since the cookie acts as your authentication. You might also want to play with the "file.readahead" parameter. We found it makes a big difference to throughput.

Now you can open the overlay file in guestfish as usual:

$ export LIBGUESTFS_BACKEND=direct
$ guestfish
><fs> add /tmp/overlay.qcow2 copyonread:true
><fs> run
><fs> list-filesystems
/dev/sda1: ext4
><fs> mount /dev/sda1 /

… and so on.

Tip: Set a root password on a disk image

virt-sysprep --enable customize --root-password password:123456 -a /dev/sdX

Useful when installing Fedora on ARM machines that only have a serial port.