Streaming NBD server

2021 update: nbdcopy is a better way to stream to and from stdin/stdout and qemu images.

The command:

qemu-img convert input output

does not work if the output is a pipe.

It’d sure be nice if it did though! For one thing, we could use this in virt-v2v to stream images into OpenStack Glance (instead of having to spool them into a temporary file).

I mentioned this to Paolo Bonzini yesterday and he suggested a simple workaround. Just replace the output with:

qemu-img convert -n input nbd:...

and write an NBD server that turns the sequence of writes from qemu-img into a stream that gets written to a pipe. Assuming the output is raw, then qemu-img convert will write, starting at disk offset 0, linearly through to the end of the disk image.

How to write such an NBD server easily? nbdkit is a project I started to make it easy to write NBD servers.

So I wrote a streaming plugin which does exactly that, in 243 lines of code.

Using a feature called captive nbdkit, you can rewrite the above command as:

nbdkit -U - streaming pipe=/tmp/output --run '
  qemu-img convert -n input -O raw $nbd
'

(This command will “hang” when you run it — you have to attach some process to read from the pipe, eg: md5sum < /tmp/output)

Further work

The streaming plugin will a lot more generally useful if it supported a sliding window, allowing limited reverse seeking and reading. So there’s a nice little project for a motivated person. See here

qemu-img diff

This patch I posted yesterday needs some reviews! It adds a qemu-img “diff” subcommand. One use for this is for when you’ve copied/cloned a guest, and you want to go back to a thin-provisioned guest over a base image.

To show it in action, we can use virt-ls. Firstly I’ll start with a Windows 2003 guest which I’ll copy:

$ xzcat /mnt/media/guest-library/win2003.img.xz > base.img
$ cp base.img copy.img

At this point, I boot copy.img, log in and install the Windows updates, just so the guest has some changes over the base image.

Now I have two 6 GB files**, but I know that they’re very similar to each other:

$ ll -h
total 6.9G
-rw-rw-r--. 1 rjones rjones 6.0G May 18 11:26 base.img
-rw-rw-r--. 1 rjones rjones 6.0G May 18 11:42 copy.img

I could save some space here using qemu-img diff!

$ ~/d/qemu/qemu-img diff -b base.img copy.img diff.qcow2
Formatting 'diff.qcow2', fmt=qcow2 size=6442450944 backing_file='base.img' encryption=off cluster_size=65536 
$ ll -h
total 7.2G
-rw-rw-r--. 1 rjones rjones 6.0G May 18 11:26 base.img
-rw-rw-r--. 1 rjones rjones 6.0G May 18 11:42 copy.img
-rw-r--r--. 1 rjones rjones 308M May 18 11:44 diff.qcow2

copy.img is no longer needed because base.img + diff.qcow2 contains the same data. Before we delete it, let’s check that using virt-ls*:

$ virt-ls -lR -a base.img --checksum / > base.files
$ virt-ls -lR -a copy.img --checksum / > copy.files
$ virt-ls -lR -a diff.qcow2 --checksum / > diff.files

Note we expect that “base” is different from the other two, but “copy” and “diff” should be identical if all went well:

$ md5sum *.files
bef46650ff6f2d3f22f562c2a820230f  base.files
fc723e43a9c16bffd6a9fb7dfd701752  copy.files
fc723e43a9c16bffd6a9fb7dfd701752  diff.files

So we can now in confidence delete copy.img.

* = Note there is a bug which stops --checksum from working in released versions of virt-ls.

** = The eagle-eyed will spot that I don’t in fact have two 6 GB files, because cp has cleverly made the copy sparse, Nevertheless qemu-img diff still saves space.

Examine VMWare ESX with libguestfs

We worked out yesterday (thanks Matthias Bolte) how to use the libguestfs tools like virt-inspector and guestfish to examine VMWare VMs.

VMWare’s native disk format is VMDK, which is only partially understood by free tools like qemu-img. qemu-img breaks quite badly on the newer variant that ESX 4 uses. Then there’s the issue of how you get the VMDK file from VMWare’s storage. Use their proprietary storage APIs?

Well it turned out both problems could be solved easily. VMWare ESX servers make the storage available over https connections, so you can use a URL like https://root:password@esxserver/folder/ to browse available storage on the server. And VMWare also makes the raw (“flat”) disk images available in the same way.

libvirt has supported ESX management for a while (thanks again to Matthias Bolte), so you can do:

$ sudo virsh -c esx://192.168.2.121/?no_verify=1 list --all
Enter username for 192.168.2.121 [root]:
Enter root password for 192.168.2.121:
 Id Name                 State
----------------------------------
  - TestLinux            shut off
  - TestWin              shut off

(Note that the domains must be shut off before VMWare will allow you to access the flat disk images).

Then we can get the storage URL:

$ sudo virsh -c esx://192.168.2.121/?no_verify=1 dumpxml TestLinux > /tmp/xml
$ grep '<source file' /tmp/xml
      <source file='[Storage1] TestLinux/TestLinux.vmdk'/>

And from that storage URL you can grab the disk image directly:

$ wget --no-check-certificate 'https://root:password@192.168.2.121/folder/TestLinux/TestLinux-flat.vmdk?dcPath=ha-datacenter&dsName=Storage1'

The large flat file downloaded is a straight raw disk image that can be examined directly in programs like guestfish and virt-inspector:

$ virt-list-filesystems -al TestLinux-flat.vmdk
/dev/sda1 ext4
/dev/vg_testlinux/lv_root ext4
/dev/vg_testlinux/lv_swap swap

$ guestfish --ro -a TestLinux-flat.vmdk -m /dev/vg_testlinux/lv_root

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help with commands
      'quit' to quit the shell

><fs> ll /
total 116
dr-xr-xr-x.  23 root root  4096 Dec 30 06:27 .
dr-xr-xr-x   29 root root     0 Dec 21 07:59 ..
-rw-r--r--.   1 root root     0 Dec 30 06:27 .autofsck
drwx------.   3 root root  4096 Dec 17 11:58 .dbus
-rw-r--r--.   1 root root     0 Dec 17 12:50 .readahead_collect
dr-xr-xr-x.   2 root root  4096 Dec 17 12:11 bin
[etc]

It’s probably also possible to avoid the download step, since libguestfs is built on qemu which should support http(s) connections directly, but I didn’t try this yet.

Use a snapshot to test libguestfs changes

If you want to test out some experimental libguestfs changes to a virtual machine, you can use a qemu snapshot to test changes before committing them. Here’s how.

First, use qemu-img to create a snapshot. In this case I have a base file called “RHEL52PV32-20090213.img” and my temporary snapshot will just be called “snapshot”. The file “snapshot” is linked to the base file.

$ qemu-img create -b RHEL52PV32-20090213.img -f qcow2 snapshot
Formatting 'snapshot', fmt=qcow2, backing_file=RHEL52PV32-20090213.img, size=8388608 kB

Now we start guestfish on the snapshot. Notice a new feature of guestfish – just use “guestfish -i image” or “guestfish -i libvirt_domain” to automatically mount all the partitions within the image in the correct places. In this case I wanted to remove the root password from the image:

$ guestfish -i snapshot

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help with commands
      'quit' to quit the shell

><fs> mounts
/dev/mapper/VolGroup00-LogVol00
/dev/sda1
><fs> vi /etc/shadow
><fs> sync

At this point, only the snapshot has changed. The base image is left alone. I test-booted the snapshot to check that my change was correct. If the change was bad, I could have just deleted the snapshot file, but since everything was OK, I committed the change back into the base image:

$ qemu-img commit snapshot
Image committed.

The “snapshot” file is no longer needed, unless you want to test further changes.

(Thanks to Dan Berrange for working out the details)