nbdkit tar filter

nbdkit is our high performance liberally licensed Network Block Device server, and OVA files are a common pseudo-standard for exporting virtual machines including their disk images.

A .ova file is really an uncompressed tar file:

$ tar tf rhel.ova
rhel.ovf
rhel-disk1.vmdk
rhel.mf

Since tar files usually store their content unmangled, this opens an interesting possibility for reading (or even writing) the embedded disk image without needing to unpack the tar. You just have to work out the offset of the disk image within the tar file. virt-v2v has used this trick to save a copy when importing OVAs for years.

nbdkit has also included a tar plugin which can access a file inside a local tar file, but the problem is what if the tar file doesn’t happen to be a local file? (eg. It’s on a webserver). Or what if it’s compressed?

To fix this I’ve turned the plugin into a filter. Using nbdkit-tar-filter you can unpack even non-local compressed tar files:

$ nbdkit curl http://example.com/qcow2.tar.xz \
         --filter=tar --filter=xz tar-entry=disk.qcow2

(To understand how filters are stacked, see my FOSDEM talk from last year). Because in this example the disk inside the tarball is a qcow2 file, it appears as qcow2 on the wire, so:

$ guestfish --ro --format=qcow2 -a nbd://localhost

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: ‘help’ for help on commands
      ‘man’ to read the manual
      ‘quit’ to quit the shell

><fs> run
><fs> list-filesystems 
/dev/sda1: ext2
><fs> mount /dev/sda1 /
><fs> ll /
total 19
drwxr-xr-x   3 root root  1024 Jul  6 20:03 .
drwxr-xr-x  19 root root  4096 Jul  9 11:01 ..
-rw-rw-r--.  1 1000 1000    11 Jul  6 20:03 hello.txt
drwx------   2 root root 12288 Jul  6 20:03 lost+found

Tip: In guestfish, extract files from disk image to a local directory

You’ve loaded your filesystem in guestfish, how do you extract all the files into the local directory on the host? Easy …

><fs> tgz-out / - | tar zxf -

This is a little bit subtle. Here’s how it works:

On the left hand side of the pipe, we run the guestfish “tgz-out” command which turns / into a tarball and sends it to “-” (stdout).

On the right hand side of the pipe is a host command, tar, which unpacks stdin into the local directory. (This explains how pipes work in guestfish)

N.B.: An alternative, for VM images, is to use the higher-level virt-tar command.

libguestfs – fun with tar

I’ve added tar support to libguestfs. Two examples:

Extract a directory subtree

This pulls out /usr/share/doc from the guest and saves it as a local file called /tmp/doc.tar.gz:

$ guestfish -a guest.img -m /dev/VolGroup00/LogVol00 

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help with commands
      'quit' to quit the shell

><fs> tgz-out /usr/share/doc /tmp/doc.tar.gz

Convert a tarball to an ext3 filesystem

$ cat /tmp/tar2ext3.sh
#!/bin/sh -
if [ $# -ne 2 ]; then
    echo "tar2ext3 input.tgz output.img"; exit 1
fi
guestfish <<EOF
alloc $2 100M
run
sfdisk /dev/sda 0 0 0 ,
mkfs ext3 /dev/sda1
mount /dev/sda1 /
tgz-in $1 /
sync
EOF
$ /tmp/tar2ext3.sh libguestfs-1.0.2.tar.gz test.img
$ ll test.img
-rw-rw-r--. 1 rjones rjones 104857600 2009-04-20 15:49 test.img

Now test.img contains all those files, inside an ext3 filesystem.