Tag Archives: network block device

nbdkit new eval plugin and ip filter

nbdkit is our flexible toolkit for building block devices. I just added a couple of new features which will appear in the next stable release, nbdkit 1.18.

Previously I’ve talked on this blog and gave a talk at FOSDEM about how you can write block devices in shell script using nbdkit-sh-plugin. But that requires you to use an extra file for the script. What if opening an extra file is too much work? Well now you can specify the script directly on the nbdkit command line using the new eval plugin.

You can write code like:

nbdkit eval \
       config='ln -sf "$(realpath "$3")" $tmpdir/file' \
       get_size='stat -Lc %s $tmpdir/file' \
       pread='dd if=$tmpdir/file skip=$4 count=$3 iflag=count_bytes,skip_bytes' \
       pwrite='dd of=$tmpdir/file seek=$4 conv=notrunc oflag=seek_bytes' \
       file=disk.img

which is a complete NBD server / block device backed by a local file. Of course it’s probably easier to use nbdkit-file-plugin for this, but the shell script gives you more control like letting you simulate failures or delays.

The other new feature is connected to a CVE we had earlier this year. CVE-2019-14850 happened because nbdkit used to open the plugin as soon as any client established a TCP connection. For some plugins opening them is quite a heavyweight action (eg. it might mean that the plugin has to establish a connection to a second server). This is before NBD negotiation or TLS had started, and it allowed clients potentially to overwhelm the server with requests even if those clients would not be authorized to connect.

To fix this we delay opening plugins until after the NBD handshake (and thus TLS authentication) has completed. But this in turn meant there was no way for plugins to reject connections early, for example based on IP address. So now I have added a preconnect method which gets runs on first TCP connection and can be used to do lightweight early filtering. There is a new nbdkit-ip-filter which implements simple TCP-wrappers-style allow/deny lists.

Leave a comment

Filed under Uncategorized

Short talk about NBD from the KVM Forum 2019

Here’s our short talk about Network Block Device (NBD) given at the KVM Forum last month:

1 Comment

Filed under Uncategorized

nbdkit as a flexible alternative to loopback mounts

Look:

$ nbdkit -f -v memory size=$(( 2**63-1 ))

On the same machine:

# modprobe nbd
# nbd-client localhost /dev/nbd0
Warning: the oldstyle protocol is no longer supported.
This method now uses the newstyle protocol with a default export
Negotiation: ..size = 8796093022207MB
Connected /dev/nbd0
# sgdisk -n 1 /dev/nbd0
Creating new GPT entries in memory.
The operation has completed successfully.
# gdisk -l /dev/nbd0
Number  Start (sector)    End (sector)  Size       Code  Name
   1            1024  9007199254740973   8.0 EiB     8300  

What’s happening here is we’re creating an 8 exabyte disk backed by an nbdkit instance which is storing the data in a sparse array in memory.

You can then try fun things like creating massive XFS filesystems.

A few caveats:

I think it’d be interesting to integrate this into filesystem test suites. Unfortunately use of the Linux NBD kernel driver needs root 😦

8 Comments

Filed under Uncategorized

NBD with TLS-PSK

The Network Block Device (NBD) protocol is really useful to us when we deal with virtual machines and disk images. It lets us share disk images between machines and is also the universal protocol we use for communicating disk images between different bits of software. I wrote a pluggable NBD server called nbdkit to make this even easier.

However there was a problem: The protocol has no concept of logins. If you have an open NBD port, then anyone can connect and read or write your disk image. This is not quite as terrible as it sounds since when two processes are talking NBD to each other, we use a Unix domain socket and we hide the socket in a directory with restrictive permissions. But there are still cases — such as communicating between separate servers — where authentication would be useful.

NBD does let you upgrade the protocol to use TLS, and all the important NBD servers support that. You can use TLS to do client authentication but it’s seriously clunky and difficult to set up because you have to use X.509 certificates, and if we’ve learned anything from the web we know that X.509 is a plot by the NSA to stop us using encryption (only joking, spooks!)

It turns out there’s a more sensible corner of the TLS specification called TLS-PSK. This uses usernames and randomly generated Pre-Shared Keys (PSK). As long as you can ensure that both the client and server can read a simple username:key file of keys, and the keys are kept secret, you can both authenticate and communicate securely.

Unfortunately just implementing TLS doesn’t get you PSK as well, and no existing NBD server supports TLS-PSK.

So I had to add support. To qemu and qemu-nbd. And to nbdkit.

Amazingly it all works, and qemu and nbdkit interoperate too. Here’s how you could use it:

$ mkdir -m 0700 /tmp/keys
$ psktool -u rich -p /tmp/keys/keys.psk
$ nbdkit -n \
    --tls=require --tls-psk=/tmp/keys/keys.psk \
    file file=disk.img
$ qemu-img info \
    --object "tls-creds-psk,id=tls0,endpoint=client,username=rich,dir=/tmp/keys" \
    --image-opts "file.driver=nbd,file.host=localhost,file.port=10809,file.tls-creds=tls0"

The qemu command line is a bit clunky, but it’s overall much simpler than setting up certificates, although not as scalable for large installations.

3 Comments

Filed under Uncategorized

New in nbdkit: Run nbdkit as a captive process

New in nbdkit ≥ 1.1.6, you can run nbdkit as a “captive process” under external programs like qemu or guestfish. This means that nbdkit runs for as long as qemu/guestfish is running, and when they exit it cleans up and exits too.

Here is a rather involved way to boot a Fedora 20 guest:

$ virt-builder fedora-20
$ nbdkit file file=fedora-20.img \
    --run 'qemu-kvm -m 1024 -drive file=$nbd,if=virtio'

The --run parameter is what tells nbdkit to run as a captive under qemu-kvm. $nbd on the qemu command line is substituted automatically with the right nbd: URL for the port or socket that nbdkit listens on. As soon as qemu-kvm exits, nbdkit is killed and cleaned up.

Here is another example using guestfish:

$ nbdkit file file=fedora-20.img \
    --run 'guestfish --format=raw -a $nbd -i'

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

Operating system: Fedora release 20 (Heisenbug)
/dev/sda3 mounted on /
/dev/sda1 mounted on /boot

><fs>

The main use for this is not to run the nbdkit file plugin like this, but in conjunction with perl and python plugins, to let people easily open and edit OpenStack Glance/Cinder and other unconventional disk images.

4 Comments

Filed under Uncategorized

New in nbdkit: Write plugins in Python

nbdkit is a permissively licensed Network Block Device server that lets you export “unusual” disk sources to qemu and libguestfs.

New in nbdkit 1.1.5, you can write plugins using Python. Here is an example.

1 Comment

Filed under Uncategorized