Streaming NBD server

2021 update: nbdcopy is a better way to stream to and from stdin/stdout and qemu images.

The command:

qemu-img convert input output

does not work if the output is a pipe.

It’d sure be nice if it did though! For one thing, we could use this in virt-v2v to stream images into OpenStack Glance (instead of having to spool them into a temporary file).

I mentioned this to Paolo Bonzini yesterday and he suggested a simple workaround. Just replace the output with:

qemu-img convert -n input nbd:...

and write an NBD server that turns the sequence of writes from qemu-img into a stream that gets written to a pipe. Assuming the output is raw, then qemu-img convert will write, starting at disk offset 0, linearly through to the end of the disk image.

How to write such an NBD server easily? nbdkit is a project I started to make it easy to write NBD servers.

So I wrote a streaming plugin which does exactly that, in 243 lines of code.

Using a feature called captive nbdkit, you can rewrite the above command as:

nbdkit -U - streaming pipe=/tmp/output --run '
  qemu-img convert -n input -O raw $nbd

(This command will “hang” when you run it — you have to attach some process to read from the pipe, eg: md5sum < /tmp/output)

Further work

The streaming plugin will a lot more generally useful if it supported a sliding window, allowing limited reverse seeking and reading. So there’s a nice little project for a motivated person. See here


Filed under Uncategorized

5 responses to “Streaming NBD server

  1. Guest

    If `qemu-img convert` writes linearly starting at 0 then how about:

    qemu-img convert input /dev/stdout | whatever

    • rich

      Except that doesn’t work:

      $ qemu-img convert sda.img /dev/stdout 
      qemu-img: Could not open '/dev/stdout': Could not refresh total sector count: Illegal seek

      The reasons are because of the way qemu block drivers work, and it could be worked around (at least, in the raw case) by adding a special case path to qemu-img, but that code isn’t written yet, and the approach of special casing is controversial.

  2. Pingback: 大変役に立ったページへのリンク集 | haruo31's blog

  3. This is pretty cool, but what if the input is a stream? My use case is I’m pulling the input image over a network, and since there is authentication involved I can’t pass the URL into qemu-img directly. I also want to calculate a check sum to guard against corruption during the transfer. Since the network is the bottleneck, working with chunks of streaming data will be much faster than sequential discrete file operations.

    Specifically I’d like to do something like curl MyURL | tee >(md5sum > input.md5) | qemu-img convert -p /dev/stdin -O output.img

    I might be able to do away with the checksum if I do the copy over HTTPS, but that doesn’t solve the problem.

    • rich

      So I’ll tell you how virt-v2v handles this, which may or may not be appropriate: We first of all make a curl connection to the source [VMware vCenter in this case], negotiate all the authentication, and capture the cookie that is passed back by the source. Then we can use qemu-img, passing the cookie to the curl driver. Since the cookie (in our case) captures that we previously authenticated, qemu-img is able to read from the source and qemu-img convert works as expected.

      Probably best to look at the code if you want the nitty gritty details:

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.