Use guestfish, virt tools with remote disks

New in libguestfs ≥ 1.21.30 is the ability to use guestfish and some of the virt tools with remote disks.

Currently you can use remote disks over NBD, GlusterFS, Ceph, Sheepdog and (recently upstream) SSH.

For this example I’ll use SSH because it needs no setup, although this requires absolutely the latest qemu and libguestfs (both from git).

Since we don’t have libvirt support for ssh yet, so this only works with the direct backend:

$ export LIBGUESTFS_BACKEND=direct

I can use a ssh:// URI to add disks with guestfish, guestmount and most of the virt tools. For example:

$ virt-rescue -a ssh://localhost/tmp/f17x64.img
[... lots of boot messages ...]
Welcome to virt-rescue, the libguestfs rescue shell.

Note: The contents of / are the rescue appliance.
You have to mount the guest's partitions under /sysroot
before you can examine them.

><rescue> mount /dev/vg_f17x64/lv_root /sysroot
><rescue> cat /sysroot/etc/redhat-release
Fedora release 17 (Beefy Miracle)

Apart from being a tiny bit slower, it just works as if the disk was local:

$ virt-df -a ssh://localhost/tmp/f17x64.img
Filesystem                           1K-blocks       Used  Available  Use%
f17x64.img:/dev/sda1                    487652      63738     398314   14%
f17x64.img:/dev/vg_f17x64/lv_root     28316680    4285576   22586036   16%

$ guestmount -a ssh://localhost/tmp/f17x64.img -i /tmp/mnt
$ ls /tmp/mnt
bin   dev  home  lib64       media  opt   root  sbin  sys  usr
boot  etc  lib   lost+found  mnt    proc  run   srv   tmp  var
$ cat /tmp/mnt/etc/redhat-release
Fedora release 17 (Beefy Miracle)
$ guestunmount /tmp/mnt

Cool new bash-completions of libguestfs tools

Starting in libguestfs ≥ 1.21.23-2, bash tab completions of guestfish, guestmount and virt-* tools have been rewritten and greatly improved.

Note you will need to install the libguestfs-bash-completion package to enable this feature.

You can now tab complete all long options on most tools:

$ virt-df --[tab]
--add             --domain          --human-readable  --uuid
--connect         --format          --inodes          --verbose
--csv             --help            --one-per-guest   --version
$ virt-resize --[tab]
--align-first          --help                 --no-extra-partition
--alignment            --ignore               --ntfsresize-force
--debug                --lvexpand             --output-format
--debug-gc             --lv-expand            --quiet
--delete               --LVexpand             --resize
--dryrun               --LV-expand            --resize-force
--dry-run              --machine-readable     --shrink
--expand               --no-copy-boot-loader  --version
--format               --no-expand-content    

Where appropriate, the -d option will now expand to the list of libvirt domains:

# virt-df -d [tab]
archlinux20121201x64  f19rawhidex32
f18x64                f19rawhidex64

Finally, guestfish commands are expanded on the command line:

$ guestfish add /tmp/disk : run : list-[tab]
list-9p              list-events          list-md-devices
list-devices         list-filesystems     list-partitions
list-disk-labels     list-ldm-partitions  
list-dm-devices      list-ldm-volumes     

To make this less intrusive, so you can really use it daily, I left the default readline expansions enabled. This means that filenames and so on can continue to be used in every position on the command line, and should mean that bash completions won’t try to be cleverer than the user.

Libguestfs bash completions are also demand-loaded now, so that if you’re not using them, they don’t consume any resources in the shell.

New in libguestfs 1.21.15: parallel virt-df and virt-alignment-scan

New in libguestfs ≥ 1.21.15, the virt-df and virt-alignment-scan tools now use parallel appliances when scanning your libvirt guests.

The amount of parallelism is selected heuristically when the tool starts up — by dividing the amount of free memory in MB by 500. You can also override this choice by using the new -P option to both tools, but the default should be fine for everyone. -P 1 disables multiple threads.

Users won’t see much difference, although I found that both tools are noticeably faster.

The implementation of threads in these tools is a little bit interesting. Of course there is a pool of worker threads. These take the libvirt guests from a list sorted in alphabetical order and process them.

However each guest takes a variable amount of time to process, and the trick is that the output from each thread mustn’t overlap or be in non-alphabetical order.

The worker threads do two things to ensure this: Firstly output from each guest scan is saved up in an open_memstream buffer. Secondly, domains are retired in order using a pthread condition variable — each worker waits until the previous domain has been retired, before retiring (ie. printing) its own result.

The outcome is that there should be no difference between what the old tools and the rewritten tools print out.

Hotplugging in libguestfs

I just posted a new set of patches upstream which add hotplugging support for libguestfs.

Since the project started, most libguestfs programs have looked like this:

/* create the handle */
g = guestfs_create ();
/* add 1 or more disks to examine */
guestfs_add_drive (g, "disk.img");
/* launch the handle */
guestfs_launch (g);

It was not possible to add further disks after launch, because of the way libguestfs works: You could not add disks to the small appliance that we use after the appliance has started up.

Except that you can: Linux and qemu have supported hotplugging disks for a long time, but we didn’t expose this through libguestfs.

Now that you can use libvirt to run the appliance it becomes relatively easy for us to add hotplugging, implemented via libvirt’s virDomainAttachDevice API. (The actual implementation inside libvirt is very complex, hence the reason why we didn’t try to reimplement it in libguestfs). From the point of view of libguestfs callers, you can now call the add-drive* APIs after launch.

All good news? Apart from the obvious limitation that you have to be using the libvirt backend for it to work at all, is it a good idea to use hotplugging?

From a security point of view, there is a trade-off: If you have to modify lots of guests, it’s faster to use hotplugging because you save having to start a new appliance for each guest. But it’s also less secure (significantly so) because one guest may be able to exploit and hijack the appliance and interfere with the disks of other appliances.

In virt-df we accept this risk. It is largely mitigated because the disks are read-only so an exploit from one guest cannot make a permanent change to any other guest.

But if you are considering using hotplugging to modify lots of mutually untrustworthy disks, don’t do it. Either use one appliance per group of mutually trusting guests, or (easier) just launch an appliance per guest. (Reading this page may help you to get the best performance in this case).

New in libguestfs 1.12: use UUIDs in virt tools

In this series of posts I’ll be looking at what’s new in the forthcoming release of libguestfs 1.12.

If you follow the example in my Summit handout you can use virt-df to monitor your VMs’ disk usage over time. But what if your VMs migrate between hosts or get renamed? Previously you had to rely on the guest name being unique and not changing. In libguestfs 1.12 we have changed most of the virt tools so you can now use a UUID to specify guests instead of a name.

# virt-df -d 799ae7ca-23a2-f622-c691-48feebcddbe6 -h
Filesystem                      Size       Used  Available  Use%
799ae7ca-23a2-f622-c691-48feebcddbe6:/dev/sda1
                                484M        67M       393M   14%
799ae7ca-23a2-f622-c691-48feebcddbe6:/dev/vg_f13x64/lv_root
                                7.4G       4.3G       3.1G   58%

Tip: Use libguestfs on VMware ESX guests

You can use libguestfs, guestfish and the virt tools on VMware ESX guests quite easily. However it’s not obvious how to do it, so this post explains that.

You will need:

• libguestfs tools installed on a Linux machine
• sshfs installed on the same Linux machine
• ssh access to the VMware ESX storage (find the root password from the administrator)
• the name of the guest and the name of the storage volume that the guest is stored on

The guest must be shut down (more on this later).

First of all, make sure you are able to ssh as root to the VMware ESX storage. It will look something like this:

$ ssh root@vmware
root@vmware's password: ****
Last login: Wed May  4 20:47:50 2011 from [...]
[root@vmware ~]# ls -l /vmfs/
total 1
drwxr-xr-x 1 root root 512 May 10 09:22 devices
drwxr-xr-x 1 root root 512 May 10 09:22 volumes

Now you should create a temporary mount point, and mount /vmfs from the VMware ESX storage server using sshfs. The command is quite simple and you don’t need to be root on the Linux side:

$ mkdir /tmp/vmfs
$ sshfs root@vmware:/vmfs /tmp/vmfs
root@vmware's password: ****
$

In another window you can navigate to the guest. For example if the guest was called “test” and it lived on volume “Storage1” then:

$ cd /tmp/vmfs/volumes/Storage1/test
$ ls -l
total 1718720
-rw------- 1 root root 8589934592 May 10 09:48 test-flat.vmdk
-rw------- 1 root root       8684 May 10 09:37 test.nvram
-rw------- 1 root root        469 Apr  4 08:16 test.vmdk
-rw------- 1 root root          0 May 11  2010 test.vmsd
-rwxr-xr-x 1 root root       2666 May 10 09:37 test.vmx
-rw------- 1 root root        259 May 11  2010 test.vmxf
-rw-r--r-- 1 root root      53966 May 11  2010 vmware-1.log
-rw-r--r-- 1 root root      78771 May 11  2010 vmware-2.log
-rw-r--r-- 1 root root      56483 Apr  4 08:15 vmware-3.log
-rw-r--r-- 1 root root      56305 May 10 09:37 vmware.log

The critical file is guestname-flat.vmdk which is the flat disk image. You can just open this for read or write using guestfish, virt-df, virt-filesystems or other libguestfs tools or programs.

For example:

$ guestfish --rw -i -a test-flat.vmdk

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

Operating system: Red Hat Enterprise Linux Server release 5.5 (Tikanga)
/dev/VolGroup00/LogVol00 mounted on /
/dev/vda1 mounted on /boot

><fs> touch /tmp/hello
><fs> ll /tmp
total 20
drwxrwxrwt.  3 root root 4096 May 10 14:48 .
drwxr-xr-x. 24 root root 4096 May 10 14:36 ..
drwxrwxrwt   2 root root 4096 Apr  4 13:16 .ICE-unix
-rw-r--r--   1 root root    0 May 10 14:48 hello

Notice that guestfish determined the guest operating system and lets you edit the disk.

$ virt-filesystems -a test-flat.vmdk --all --long -h
Name                     Type       VFS  Label Size Parent
/dev/sda1                filesystem ext3 /boot 102M -
/dev/VolGroup00/LogVol00 filesystem ext3 -     7.1G -
/dev/VolGroup00/LogVol01 filesystem swap -     768M -
/dev/VolGroup00/LogVol00 lv         -    -     7.1G /dev/VolGroup00
/dev/VolGroup00/LogVol01 lv         -    -     768M /dev/VolGroup00
/dev/VolGroup00          vg         -    -     7.9G -
/dev/sda2                pv         -    -     7.9G -
/dev/sda1                partition  -    -     102M /dev/sda
/dev/sda2                partition  -    -     7.9G /dev/sda
/dev/sda                 device     -    -     8.0G -

$ virt-df -a test-flat.vmdk -h
Filesystem                                Size       Used  Available  Use%
test-flat.vmdk:/dev/sda1                   99M        12M        81M   13%
test-flat.vmdk:/dev/VolGroup00/LogVol00
                                          6.9G       1.1G       5.5G   16%

With libguestfs we usually allow you to read guests which are running. The results might be inconsistent at times, but it generally works. However VMware itself doesn’t allow running guests to be read. If the guest is running you can see that VMware prevents access:

# file test-flat.vmdk
test-flat.vmdk: writable, regular file, no read permission

Whereas when the same guest is shut down, reads (and writes) are allowed:

# file test-flat.vmdk
test-flat.vmdk: x86 boot sector; partition 1: ID=0x83, active, starthead 1, startsector 63, 208782 sectors; partition 2: ID=0x8e, starthead 0, startsector 208845, 16563015 sectors, code offset 0x48

This is a limitation of VMware and nothing to do with libguestfs.

A note on performance: I run this from my home to a VMware server which is a third of the way around the planet over plain 2Mbps ADSL. It’s noticeably slower than accessing local disk images, but still very usable. sshfs appears to be very efficiently implemented. It is far faster and more convenient than copying the whole disk image around.

Changes ahead for libguestfs RHEL 6.1 package

I previously said that libguestfs in RHEL 6.1 would be based on the recent upstream 1.6 release.

This plans have had to change slightly. It looks like we’ll rebase to 1.7.16 (a development version).

The reason is simply that to get into the next release of RHEV we had to remove the Perl dependencies on a number of key programs, because the tiny RHEV-H hypervisor [PDF] doesn’t have space to include Perl. Several programs like virt-inspector and virt-df had to be rewritten in C. We could backport all of the changes but they amount to nearly every change since 1.6 anyway.

What I do have to do is to meticulously check each C program precisely matches the old Perl version, in terms of output, command line arguments and so on, so that scripts written against RHEL 6.0 won’t break. But that’s what you pay Red Hat for.

Preview packages will be available here.

virt-tools.org: Advanced virt-df tips

Read the advanced tips for using virt-df, including importing the results into a PostgreSQL database, making graphs, and getting email alerts.

libguestfs binaries for all Linux distros

I uploaded some distro-independent Linux/x86-64 binaries of libguestfs, guestfish, guestmount and the virt-* tools. Be sure to read the README file first.

These are a little experimental and I’d welcome feedback. I got them to work fine on Debian 5.0 after upgrading glibc and Perl, but YMMV.

Update: OpenSUSE 11.3 working.

The version of KVM supplied is too old (doesn’t support virtio-serial) so I had to compile qemu from git and drop the following qemu wrapper in place:

#!/bin/sh -
qemudir=/home/rjones/d/qemu
exec $qemudir/x86_64-softmmu/qemu-system-x86_64 -L $qemudir/pc-bios "$@"

The next problem which had me confused for a very long time was that qemu kept aborting while allocating memory. After trying things like adding swap, playing with overcommit settings and so on, it turned out that SUSE uses some really silly, and very small default ulimits for virtual memory. You have to do:

ulimit -Hv unlimited
ulimit -Sv unlimited
ulimit -Hm unlimited
ulimit -Sm unlimited

Update: Ubuntu 10.04 working.

As with SUSE, the version of KVM shipped by Ubuntu is too old to support virtio-serial. All I had to do was replace it with qemu from git and the same qemu wrapper above.

After that guestfish works fine.

If you need the Perl tools, then you have to upgrade Perl to 5.12.1, install hivex, and link libpcre.so.0 -> libpcre.so.3.

Be script friendly in your tools output

If your command line tool prints something, make it friendly to shell scripts. This example was sent to me last week:

virsh # pool-list --details --all
Name       State     Autostart  Persistent  Capacity  Allocation  Available
---------------------------------------------------------------------------
default    running   yes        yes          1.79 TB     1.49 TB  304.77 GB
image_dir  running   yes        yes          1.79 TB     1.49 TB  304.77 GB
tmp        inactive  no         yes                -           -          -

And this was my reply:

One good thing, and several bad things about that. The good thing is
that empty columns are presented with ‘-‘ which means you can use awk
and sort -k to parse the output columnwise.

The bad things:

1. Space within fields “1.79 TB” (awk / sort -k in fact won’t work).
2. Numeric fields aren’t numbers: You can’t sort -n on “1.79 TB”, and you can’t read that number into a script and do math on it. Most tools have a “-h” or “–human” option in order to generate human-readable numbers (without spaces), but default to just printing the raw numbers.
3. Unnecessary “——-” line.
4. Title line should be optional. Have a –no-title option or something like that to suppress it.
5. Does virsh still print an unnecessary blank line after the output? If so, stop doing that.

In tools like virt-df we also provide CSV format output. CSV is a fine format for two-dimensional output, but beware the caveats.