nbdkit tar filter

nbdkit is our high performance liberally licensed Network Block Device server, and OVA files are a common pseudo-standard for exporting virtual machines including their disk images.

A .ova file is really an uncompressed tar file:

$ tar tf rhel.ova
rhel.ovf
rhel-disk1.vmdk
rhel.mf

Since tar files usually store their content unmangled, this opens an interesting possibility for reading (or even writing) the embedded disk image without needing to unpack the tar. You just have to work out the offset of the disk image within the tar file. virt-v2v has used this trick to save a copy when importing OVAs for years.

nbdkit has also included a tar plugin which can access a file inside a local tar file, but the problem is what if the tar file doesn’t happen to be a local file? (eg. It’s on a webserver). Or what if it’s compressed?

To fix this I’ve turned the plugin into a filter. Using nbdkit-tar-filter you can unpack even non-local compressed tar files:

$ nbdkit curl http://example.com/qcow2.tar.xz \
         --filter=tar --filter=xz tar-entry=disk.qcow2

(To understand how filters are stacked, see my FOSDEM talk from last year). Because in this example the disk inside the tarball is a qcow2 file, it appears as qcow2 on the wire, so:

$ guestfish --ro --format=qcow2 -a nbd://localhost

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: ‘help’ for help on commands
      ‘man’ to read the manual
      ‘quit’ to quit the shell

><fs> run
><fs> list-filesystems 
/dev/sda1: ext2
><fs> mount /dev/sda1 /
><fs> ll /
total 19
drwxr-xr-x   3 root root  1024 Jul  6 20:03 .
drwxr-xr-x  19 root root  4096 Jul  9 11:01 ..
-rw-rw-r--.  1 1000 1000    11 Jul  6 20:03 hello.txt
drwx------   2 root root 12288 Jul  6 20:03 lost+found

What is guest inspection

[This article expands on what I wrote here.]

I’ve been asked a few times if there is a standard for describing or inspecting virtual machines. Or conversely, should virt-inspector produce Enterprise Standard XML and not the ad-hoc XML that it produces right now?

Consider first what does “operating system version” mean? “Fedora 14” — what does that mean? It could mean a “class” of operating systems, waiting to be installed. Or perhaps it means a particular installed instance, together with all the configuration tweaks made by the administrator? Or perhaps you mean a bare installation before configuration (what the cool kids are now calling “JEOS”).

For a class of operating systems, before installation, what you would like to know includes:

1. kernel type (“linux”)
2. operating system name (“Fedora”)
3. operating system version (“14”)
4. architecture
5. install location
6. how to download and install it
7. do installed versions need licensing, like Windows Activation or RHN subs?
8. what preferred devices should we expose to new instances? (eg. does this have drivers for virtio, or should we expose some legacy devices?)
9. minimum and recommended resources (RAM, disk space etc)

There is no comprehensive store of this data at the moment, although virt-install hard-codes some of it. So does VMware.

An installed instance of an OS is quite different. You’d like to know:

1. what configuration changes were made?
2. what applications are installed?
3. what device drivers does it expect?
4. what’s the state of critical security fixes?

and more. virt-inspector covers some of this. It also crosses over with inventory management standards like the impenetrable CIM. It also enters configuration management territory, so tools like puppet, kickstart and cfengine are applicable.

A running OS contains even more information beyond what is in the disk image:

1. how many vCPUs?
2. how much RAM
3. what actual devices and resources?

Here we overlap with OVF, libvirt XML and more.

So there you have it: a difficult, multi-faceted, ill-defined problem with no easy answers.