Tag Archives: network block device

nbdkit as a flexible alternative to loopback mounts

Look:

$ nbdkit -f -v memory size=$(( 2**63-1 ))

On the same machine:

# modprobe nbd
# nbd-client localhost /dev/nbd0
Warning: the oldstyle protocol is no longer supported.
This method now uses the newstyle protocol with a default export
Negotiation: ..size = 8796093022207MB
Connected /dev/nbd0
# sgdisk -n 1 /dev/nbd0
Creating new GPT entries in memory.
The operation has completed successfully.
# gdisk -l /dev/nbd0
Number  Start (sector)    End (sector)  Size       Code  Name
   1            1024  9007199254740973   8.0 EiB     8300  

What’s happening here is we’re creating an 8 exabyte disk backed by an nbdkit instance which is storing the data in a sparse array in memory.

You can then try fun things like creating massive XFS filesystems.

A few caveats:

I think it’d be interesting to integrate this into filesystem test suites. Unfortunately use of the Linux NBD kernel driver needs root 😦

Advertisements

8 Comments

Filed under Uncategorized

NBD with TLS-PSK

The Network Block Device (NBD) protocol is really useful to us when we deal with virtual machines and disk images. It lets us share disk images between machines and is also the universal protocol we use for communicating disk images between different bits of software. I wrote a pluggable NBD server called nbdkit to make this even easier.

However there was a problem: The protocol has no concept of logins. If you have an open NBD port, then anyone can connect and read or write your disk image. This is not quite as terrible as it sounds since when two processes are talking NBD to each other, we use a Unix domain socket and we hide the socket in a directory with restrictive permissions. But there are still cases — such as communicating between separate servers — where authentication would be useful.

NBD does let you upgrade the protocol to use TLS, and all the important NBD servers support that. You can use TLS to do client authentication but it’s seriously clunky and difficult to set up because you have to use X.509 certificates, and if we’ve learned anything from the web we know that X.509 is a plot by the NSA to stop us using encryption (only joking, spooks!)

It turns out there’s a more sensible corner of the TLS specification called TLS-PSK. This uses usernames and randomly generated Pre-Shared Keys (PSK). As long as you can ensure that both the client and server can read a simple username:key file of keys, and the keys are kept secret, you can both authenticate and communicate securely.

Unfortunately just implementing TLS doesn’t get you PSK as well, and no existing NBD server supports TLS-PSK.

So I had to add support. To qemu and qemu-nbd. And to nbdkit.

Amazingly it all works, and qemu and nbdkit interoperate too. Here’s how you could use it:

$ mkdir -m 0700 /tmp/keys
$ psktool -u rich -p /tmp/keys/keys.psk
$ nbdkit -n \
    --tls=require --tls-psk=/tmp/keys/keys.psk \
    file file=disk.img
$ qemu-img info \
    --object "tls-creds-psk,id=tls0,endpoint=client,username=rich,dir=/tmp/keys" \
    --image-opts "file.driver=nbd,file.host=localhost,file.port=10809,file.tls-creds=tls0"

The qemu command line is a bit clunky, but it’s overall much simpler than setting up certificates, although not as scalable for large installations.

3 Comments

Filed under Uncategorized

New in nbdkit: Run nbdkit as a captive process

New in nbdkit ≥ 1.1.6, you can run nbdkit as a “captive process” under external programs like qemu or guestfish. This means that nbdkit runs for as long as qemu/guestfish is running, and when they exit it cleans up and exits too.

Here is a rather involved way to boot a Fedora 20 guest:

$ virt-builder fedora-20
$ nbdkit file file=fedora-20.img \
    --run 'qemu-kvm -m 1024 -drive file=$nbd,if=virtio'

The --run parameter is what tells nbdkit to run as a captive under qemu-kvm. $nbd on the qemu command line is substituted automatically with the right nbd: URL for the port or socket that nbdkit listens on. As soon as qemu-kvm exits, nbdkit is killed and cleaned up.

Here is another example using guestfish:

$ nbdkit file file=fedora-20.img \
    --run 'guestfish --format=raw -a $nbd -i'

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

Operating system: Fedora release 20 (Heisenbug)
/dev/sda3 mounted on /
/dev/sda1 mounted on /boot

><fs>

The main use for this is not to run the nbdkit file plugin like this, but in conjunction with perl and python plugins, to let people easily open and edit OpenStack Glance/Cinder and other unconventional disk images.

3 Comments

Filed under Uncategorized

New in nbdkit: Write plugins in Python

nbdkit is a permissively licensed Network Block Device server that lets you export “unusual” disk sources to qemu and libguestfs.

New in nbdkit 1.1.5, you can write plugins using Python. Here is an example.

1 Comment

Filed under Uncategorized