Transactions with guestfish

I was asked a few days ago if libguestfs has a way to apply a group of changes to an image together. The question was really about transaction support — applying a group of changes and then committing them or doing a rollback, with the final image either containing all the changes or none of them.

Although libguestfs doesn’t support this, you can do it using libguestfs and the qemu-img tool together. This post shows you how.

First I use virt-builder to quickly get a test image that I can play with:

$ virt-builder fedora-20

We create an overlay which will store the changes until we decide to commit or rollback:

$ qemu-img create -f qcow2 -b fedora-20.img overlay.img

Now open the overlay and make your changes:

$ guestfish -a overlay.img -i

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

Operating system: Fedora release 20 (Heisenbug)
/dev/sda3 mounted on /
/dev/sda1 mounted on /boot

><fs> write-append /etc/issue.net \
    "THIS IS A CHANGE TO ISSUE.NET\n"
><fs> cat /etc/issue.net
Fedora release 20 (Heisenbug)
Kernel \r on an \m (\l)
THIS IS A CHANGE TO ISSUE.NET
><fs> exit

The base image (fedora-20.img) is untouched, and the overlay contains the changes we made:

$ virt-cat -a fedora-20.img /etc/issue.net
Fedora release 20 (Heisenbug)
Kernel \r on an \m (\l)
$ virt-cat -a overlay.img /etc/issue.net
Fedora release 20 (Heisenbug)
Kernel \r on an \m (\l)
THIS IS A CHANGE TO ISSUE.NET

Rollback

Rollback is pretty simple!

$ rm overlay.img

Commit

The more interesting one is how to commit the changes back to the original file. Using qemu-img you just do:

$ qemu-img commit overlay.img
Image committed.
$ rm overlay.img

The changes are now contained in the original image file:

$ virt-cat -a fedora-20.img /etc/issue.net
Fedora release 20 (Heisenbug)
Kernel \r on an \m (\l)
THIS IS A CHANGE TO ISSUE.NET

ACID

Have we discovered the ACID properties of disk images? Not quite.

Although the change is atomic (A)¹, the disk image is consistent (C) before and after the change, and the change is durable (D)², the final property is not satisfied.

There is no isolation (I). Because it is infeasible to resolve conflicts at the block layer where qemu-img operates, it would be guaranteed corruption if you tried this technique in parallel on the same disk image. The only way to make it work reliably is to serialize every operation on the disk image with a mutex.

¹ The change is only atomic if you don’t look at the backing file for the short time that qemu-img commit runs.

² Strictly speaking, you must call sync or fsync after the qemu-img commit in order for the change to be durable.

Using libguestfs to find out why a Windows guest was “hanging”

While diagnosing a bug where a Windows guest hangs at boot, I used libguestfs to find out what files were being updated on the disk. Here is how.

First of all I used virt-ls to get a listing of all the files in the guest and the last time they were updated:

$ virt-ls -lR -a /path/to/winxp.img --time-relative / | \
  grep '^-' > /tmp/files

The colums in the output file look like this:

- 0777          0 12022162 12022162 12022162 /AUTOEXEC.BAT

The three numbers in columns 4, 5 and 6 (“12022162”) are the ones we are interested in. These are the time of last access, time of last modification and time of last status change, in seconds before now (because of the --time-relative flag).

So now we’re just looking for the files where column 6 is a small number. Everything that’s been touched in the last 2 minutes, for example:

$ awk '$6 < 120' < /tmp/files
- 0777       1024       40       40       40 /Documents and Settings/rjones/NTUSER.DAT.LOG
- 0777       7414       30       30       30 /WINDOWS/Prefetch/LOGON.SCR-151EFAEA.pf
- 0777    9445376       50       50       50 /WINDOWS/SoftwareDistribution/DataStore/DataStore.edb
- 0777       8192       50       50       50 /WINDOWS/SoftwareDistribution/DataStore/Logs/edb.chk
- 0777     131072       50       50       50 /WINDOWS/SoftwareDistribution/DataStore/Logs/edb.log
- 0777     203243       49       49       49 /WINDOWS/WindowsUpdate.log
- 0777       1024       41       40       40 /WINDOWS/system32/config/SAM.LOG
- 0777     262144      645       42       42 /WINDOWS/system32/config/SECURITY
- 0777      20480       42       41       41 /WINDOWS/system32/config/SECURITY.LOG

Ah.

Looks to me like Windows Update is running.

We can confirm this easily:

$ virt-cat -a /path/to/winxp.img /WINDOWS/WindowsUpdate.log|tail
2012-02-27	19:17:57:718	 824	144	DnldMgr	Error 0x80072f78 occurred while downloading update; notifying dependent calls.
2012-02-27	19:18:12:546	 824	144	DnldMgr	Error 0x80072f78 occurred while downloading update; notifying dependent calls.
2012-02-27	19:18:39:015	 824	14c	DnldMgr	Error 0x80072f78 occurred while downloading update; notifying dependent calls.
2012-02-27	19:18:49:031	 824	7b8	DnldMgr	Error 0x80072f78 occurred while downloading update; notifying dependent calls.
2012-02-27	19:18:58:046	 824	14c	DnldMgr	Error 0x80072f78 occurred while downloading update; notifying dependent calls.
2012-02-27	19:18:58:062	 824	748	AU	AU checked download status and it changed: Downloading is paused

Indeed soon afterwards the guest came back to life, after downloading all its Windows Updates.

Tip: Another way to get the IP address of a virtual machine

In this earlier post several ways were discussed of getting the IP address of a virtual machine (specifically if the VM is acquiring a different address each time from DHCP).

Another way is to use virt-cat or virt-win-reg to grab the information out of the virtual machine itself.

When a VM acquires an IP address from DHCP then it writes the address to a log file, or in the Windows case updates the Windows Registry. The idea is simply that we’ll read the information from the VM’s log files or the Registry.

The details vary depending on the precise guest type, and probably we should wrap this up in a nice virt tool. But here’s how you would do it for Fedora/RHEL and Windows guests:

# virt-cat RHEL60x64 /var/log/messages | grep 'dhclient.*bound to'
Mar 30 19:56:22 rhel60x64 dhclient: bound to 192.168.122.220 -- renewal in 1527 seconds.
Mar 30 20:21:49 rhel60x64 dhclient: bound to 192.168.122.220 -- renewal in 1375 seconds.
Mar 30 20:44:44 rhel60x64 dhclient: bound to 192.168.122.220 -- renewal in 1287 seconds.
Mar 30 21:06:11 rhel60x64 dhclient: bound to 192.168.122.220 -- renewal in 1461 seconds.

# virt-win-reg Win7x32 \
  'HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\Tcpip\Parameters' | \
  grep DhcpIPAddress
"DhcpIPAddress"=hex(1):31,00,39,00,32,00,2e,00,31,00,36,00,38,00,2e,00,31,00,32,00,32,00,2e,00,31,00,37,00,38,00,00,00

In the Windows case you need to note that ControlSet001 isn’t always the right control set — consult this note in the virt-win-reg man page to find the correct way to do this. Furthermore virt-win-reg prints out the hex-encoded UTF16-LE string, which requires a little bit of conversion to produce a printable string (192.168.122.178 in this instance).

Update: To add to all the other methods from the previous post, and the method described above, Eric Blake also points out that nwfilter can sniff IP addresses used by VMs.

New (Year’s) libguestfs tools: virt-copy-in, virt-copy-out, virt-tar-in, virt-tar-out

One aim with libguestfs development is to make easy and common file operations easy. Although you can already upload and download files into virtual machines using guestfish commands, is there a way to make this common operation easier to discover?

One way is to add more virt commands, which I’ve found that users have least difficulty discovering because they are on the website, autocompleted when you hit virt-[tab], and listed as separated manual pages.

So today I added four more commands for uploading and downloading: virt-copy-in, virt-copy-out, virt-tar-in, virt-tar-out.

The way you use them is very simple:

$ mkdir homes
$ virt-copy-out -d Fedora14 /home homes/

$ virt-tar-out -d Fedora14 /home - | \
    gzip --best > homes.tar.gz

These commands are just small shell script wrappers around guestfish, but I hope they make common things a little bit easier.

You can get these new commands from Fedora Rawhide, or as binaries for Debian or Ubuntu.

Tip: Uploading and downloading

Talked to someone today who was confused by how to get files in and out of a virtual machine disk image. Libguestfs, guestfish, virt-cat, virt-edit, virt-ls and virt-tar offer several ways to do this, and in this article I’m going to summarize all these different ways.

First of all, answer these questions:

1. Upload (ie. from your host into the disk image) or
   download (disk image down to host)?
2. Single file or lots of files?
3. Have you got / do you want a tarball, or a directory containing the individual files?
4. Do you want to download the files or just get a list of files?

I should note as usual that uploading or modifying a guest is only safe if the guest is shut off. If you try to modify a live guest you’ll get disk corruption, so watch out.

Downloading a single file

Easiest way is with virt-cat:

virt-cat GuestName /path/to/file > file

An alternative is to use guestfish download:

guestfish -i --ro -d GuestName \
    download /path/to/file ./localfile

Uploading a single file

If the file exists and it’s a text file that you want to edit, the easiest way is to use virt-edit:

virt-edit GuestName /path/to/file

To upload a single file in general, use guestfish upload:

guestfish -i -d GuestName \
    upload ./localfile /path/to/file

Downloading multiple files

Easiest way is probably to use the copy-out command:

guestfish -i --ro -d GuestName \
    copy-out /directory ./localdir

Uploading multiple files

Use guestfish copy-in command:

guestfish -i -d GuestName <<EOF
  mkdir /directory
  copy-in ./localdir /directory
EOF

If you want to quickly upload lots of files, the fastest method is to prepare an ISO (mkisofs) or squashfs containing the files and attach it as an extra disk. You can then copy files quickly to where you need them, eg:

guestfish -i -d GuestName -a my.iso <<EOF
  mount-ro /dev/sdb /mnt
  cp-a /mnt/files /someplace
EOF

Downloading or uploading a tarball

There is a high level tool called virt-tar which automates this. You can also do it semi-manually using the guestfish commands tar-in, tar-out, tgz-in, tgz-out (and more, see the guestfish documentation).

Listing files

Finally to list out files, use the high level virt-ls tool, or the low level guestfish commands ls (flat list of a single directory) or find0 (recursive listing of subdirectories).

virt-cat now works on encrypted guests

In libguestfs 1.7.4, virt-cat can be used on encrypted guests:

# virt-cat -d F13x64Encrypted /etc/issue.net
Enter key or passphrase ("/dev/vda2"): ***
Fedora release 13 (Goddard)
Kernel \r on an \m (\l)

We’ve made a change to the way the virt-cat command line works (I’m going to make similar changes to the other libguestfs virt tools), but don’t worry because the old style is still supported and will remain supported indefinitely. The new style is less ambiguous and more flexible though.

Tip: Get the hostname of a guest

Because different operating systems store the hostname in different places, you have to know in advance what sort of OS your guest is (perhaps using virt-inspector). Perhaps we should add the hostname to virt-inspector.

This works for Fedora guests:

# virt-cat F13x64 /etc/sysconfig/network | \
  grep HOSTNAME= | \
  awk -F= '{print $2}'
f13x64.home.annexia.org

This is for Debian/Ubuntu guests:

# virt-cat Debian5x64 /etc/hostname
debian5x64.home.annexia.org

For Windows guests:

# virt-win-reg Win7x32 \
  'HKEY_LOCAL_MACHINE\System\ControlSet001\Services\Tcpip\Parameters' \
  Hostname
win7x32

I’m not completely clear how to get the DNS domain name from Windows. According to this article you should just replace “Hostname” with “Domain” in the above command, but for me that yields just an empty string.

Virtualization and whom you trust

One comment at the test day described virt-cat as “awesome but dangerous”.

This user was surprised that he could do:

virt-cat aguest /etc/shadow

and read the shadow password file from a guest. “Is there”, I was asked, “a security model for this?”

Here’s the news: You can already look at the shadow password file in any disk image using a hex editor. libguestfs, guestfish and virt-cat just make it easier.

Could you encrypt the virtual disk? That will protect the VM while it is (virtually) switched off, but as soon as you boot it up, the encryption key is stored somewhere in guest memory, and the host administrator can read that too.

No security model can help you here. You need to own and manage the hardware yourself, or you need to trust your cloud provider. If your data is at all personally or commercially sensitive, keep it on hardware you physically control.

libguestfs binaries for all Linux distros

I uploaded some distro-independent Linux/x86-64 binaries of libguestfs, guestfish, guestmount and the virt-* tools. Be sure to read the README file first.

These are a little experimental and I’d welcome feedback. I got them to work fine on Debian 5.0 after upgrading glibc and Perl, but YMMV.

Update: OpenSUSE 11.3 working.

The version of KVM supplied is too old (doesn’t support virtio-serial) so I had to compile qemu from git and drop the following qemu wrapper in place:

#!/bin/sh -
qemudir=/home/rjones/d/qemu
exec $qemudir/x86_64-softmmu/qemu-system-x86_64 -L $qemudir/pc-bios "$@"

The next problem which had me confused for a very long time was that qemu kept aborting while allocating memory. After trying things like adding swap, playing with overcommit settings and so on, it turned out that SUSE uses some really silly, and very small default ulimits for virtual memory. You have to do:

ulimit -Hv unlimited
ulimit -Sv unlimited
ulimit -Hm unlimited
ulimit -Sm unlimited

Update: Ubuntu 10.04 working.

As with SUSE, the version of KVM shipped by Ubuntu is too old to support virtio-serial. All I had to do was replace it with qemu from git and the same qemu wrapper above.

After that guestfish works fine.

If you need the Perl tools, then you have to upgrade Perl to 5.12.1, install hivex, and link libpcre.so.0 -> libpcre.so.3.

guestfish -i, that’s i for “inspector” not “interactive”

I wrote this FAQ entry today …

---

• What’s the deal with guestfish -i?
• Why does virt-cat only work on a real VM image, but virt-df works on any disk image?
• What does “no root device found in this operating system image” mean?

These questions are all related at a fundamental level which may not be immediately obvious.

At the libguestfs API and guestfish level, a “disk image” is just a pile of partitions and filesystems.

In contrast, when the virtual machine boots, it mounts those filesystems into a consistent hierarchy such as:

 /          (/dev/sda2)
 |
 +-- /boot  (/dev/sda1)
 |
 +-- /home  (/dev/vg_external/Homes)
 |
 +-- /usr   (/dev/vg_os/lv_usr)
 |
 +-- /var   (/dev/vg_os/lv_var)

(or drive letters on Windows).

The libguestfs API however knows nothing about how filesystems are mounted. In general, a disk image that we just made up might not contain an operating system or “mountpoints” at all.

Nevertheless, users expect some tools (like virt-cat) to work with VM paths:

virt-cat fedora.img /var/log/messages

How does virt-cat know that /var is a separate partition?

The trick is a large external program called virt-inspector. Virt-inspector contains all sorts of tests and heuristics, so it can take a disk image about which nothing is known beforehand and work out both what it is, and how filesystems would be mounted.

Some tools, such as virt-cat, virt-edit, virt-ls and virt-tar use the same virt-inspector code to map VM paths. Other tools, such as virt-df and virt-list-filesystems operate entirely at the raw “big pile of filesystems” level of the libguestfs API, and don’t use virt-inspector code.

guestfish is in an interesting middle ground. If you use the -a and -m command line options, then you have to tell guestfish exactly how to add disk images and where to mount partitions. This is the raw API level.

However there is also a guestfish -i (for “inspector”) mode. In this mode, guestfish runs virt-inspector, and then (if successful) virt-inspector runs guestfish a second time with the correct -a and -m options. This is much slower, but usually simpler for the user. They can just do this and it all works:

guestfish -i SomeLibvirtGuest

The error “no root device found in this operating system image” is related to this. It means virt-inspector was unable to locate an operating system within the disk image you gave it. You might see this from programs like virt-cat if you try to run them on something which is just a disk image, not a virtual machine.