You can now write nbdkit plugins in Ruby.
(So in all that makes: C/C++, Perl, Python, OCaml or Ruby as your choices for nbdkit plugins)
qemu-img convert input output
does not work if the output is a pipe.
It’d sure be nice if it did though! For one thing, we could use this in virt-v2v to stream images into OpenStack Glance (instead of having to spool them into a temporary file).
I mentioned this to Paolo Bonzini yesterday and he suggested a simple workaround. Just replace the output with:
qemu-img convert -n input nbd:...
and write an NBD server that turns the sequence of writes from qemu-img into a stream that gets written to a pipe. Assuming the output is raw, then
qemu-img convert will write, starting at disk offset 0, linearly through to the end of the disk image.
How to write such an NBD server easily? nbdkit is a project I started to make it easy to write NBD servers.
So I wrote a streaming plugin which does exactly that, in 243 lines of code.
Using a feature called captive nbdkit, you can rewrite the above command as:
nbdkit -U - streaming pipe=/tmp/output --run ' qemu-img convert -n input -O raw $nbd '
(This command will “hang” when you run it — you have to attach some process to read from the pipe, eg:
md5sum < /tmp/output)
The streaming plugin will a lot more generally useful if it supported a sliding window, allowing limited reverse seeking and reading. So there’s a nice little project for a motivated person. See here
nbdkit is a liberally licensed NBD (Network Block Device) server designed to let you connect all sorts of crazy disk images sources (like Amazon, Glance, VMware VDDK) to the universal network protocol for sharing disk images: NBD.
$ nbdkit -r curl url=http://onuma/scratch/boot.iso
and then you can read the disk image using guestfish, qemu or any other nbd client:
$ guestfish --ro -a nbd://localhost -i Welcome to guestfish, the guest filesystem shell for editing virtual machine filesystems and disk images. Type: 'help' for help on commands 'man' to read the manual 'quit' to quit the shell /dev/sda mounted on / ><fs> _
If you are using a normal SSH server like OpenSSH which supports the SSH File Transfer Protocol (aka SFTP), then you can use SFTP to access images:
$ nbdkit -r curl url=sftp://rjones@localhost/~/fedora-20.img
I’m hoping to enable write support in a future version.
It doesn’t work at the moment because I haven’t worked out how to switch between read (GET) and write (POST) requests in a single cURL handle. Perhaps I need to use two handles? The documentation is confusing.
New in nbdkit ≥ 1.1.6, you can run nbdkit as a “captive process” under external programs like qemu or guestfish. This means that nbdkit runs for as long as qemu/guestfish is running, and when they exit it cleans up and exits too.
Here is a rather involved way to boot a Fedora 20 guest:
$ virt-builder fedora-20 $ nbdkit file file=fedora-20.img \ --run 'qemu-kvm -m 1024 -drive file=$nbd,if=virtio'
--run parameter is what tells nbdkit to run as a captive under
$nbd on the qemu command line is substituted automatically with the right
nbd: URL for the port or socket that nbdkit listens on. As soon as qemu-kvm exits, nbdkit is killed and cleaned up.
Here is another example using guestfish:
$ nbdkit file file=fedora-20.img \ --run 'guestfish --format=raw -a $nbd -i' Welcome to guestfish, the guest filesystem shell for editing virtual machine filesystems and disk images. Type: 'help' for help on commands 'man' to read the manual 'quit' to quit the shell Operating system: Fedora release 20 (Heisenbug) /dev/sda3 mounted on / /dev/sda1 mounted on /boot ><fs>
The main use for this is not to run the nbdkit file plugin like this, but in conjunction with perl and python plugins, to let people easily open and edit OpenStack Glance/Cinder and other unconventional disk images.