New nbdkit “remote tmpfs” (tmpdisk plugin)

I was making some thin clients for the Fedora RISC-V project a few weeks ago. These are based on the HiFive Unleashed U540 board and so they have no local SATA, only slow, unreliable SD cards. Any filesystems that might be heavily used must be network filesystems.

As these are Linux clients we essentially have three possible choices for network filesystems: NFS, NBD or a cluster FS. As I didn’t want to set up multiple server nodes or need the redundancy, cluster filesystems are immediately discounted. NFS is — “fine”. And indeed I selected that for /home where performance is not a problem. But for the clients that are build servers I selected NBD as a high-performance block-based storage. I’m using nbdkit, the high performance flexible NBD server.

However nbdkit didn’t quite do what I wanted, so I had to write a new plugin. I needed a “remote tmpfs“.

To get a new “remote tmpfs”, a fresh filesystem each time, the client does:

modprobe nbd
nbd-client server /dev/nbd0
mount /dev/nbd0 /var/scratch

Backing this is a flexible new plugin called tmpdisk. By default it creates a new disk for each connection, formats it with mkfs and serves it to the client. But it’s also scriptable, so you can substitute any command you like instead of mkfs to create your own custom disks (I recommend looking at mke2fs -d in case you need to have pre-populated scratch disks).

It’s important to note that these are scratch disks so when the client unmounts the filesystem it is completely deleted from the server. But that’s fine for temporary directories and (for my purposes) package builds.

To find out more about nbdkit, watch my video from FOSDEM 2019.

New nbdkit data strings

You can use nbdkit, our infinitely flexible Network Block Device server to serve small disks and test images with the nbdkit data plugin. For example you can cut and paste this command into your shell to demonstrate a bootable disk image which prints “hello, world”:

nbdkit data data='
    0xb4 0 0xb0 3 0xcd 0x10 0xb4 0x13
    0xb3 0x0a 0xb0 1 0xb9 0x0e 0 0xb6
    0 0xb2 0 0xbd 0x19 0x7c 0xcd 0x10
    0xf4 0x68 0x65 0x6c 0x6c 0x6f 0x2c 0x20
    0x77 0x6f 0x72 0x6c 0x64 0x0d 0x0a
    @0x1fe 0x55 0xaa
' --run 'qemu-system-i386 -fda $nbd'

(As an aside, what is the smallest nbdkit data string that can boot to a “hello, world” message?)

The data parameter is a mini-language, and I recently extended it in an interesting way. It wasn’t possible to make repeated patterns easily before. If you wanted a disk containing 0x55 0xAA repeated (the binary bit patterns 01010101 10101010) then the only way to get that was to literally write:

nbdkit data data='0x55 0xAA 0x55 0xAA [repeated many times ...]'

but now you can group things together and write:

nbdkit data data='( 0x55 0xAA )*256'

The nesting works by recursively creating a new parser, which means you can use any data expression. For example to get 4 sectors containing half blank and half test data you can now do:

nbdkit data data='( @256 ( 0x55 0xAA )*128 )*4'

This gives you lots of way to make disks containing test patterns which you could then use to test Linux programs using /dev/nbd0 loop devices.

Goals – an experimental new tool which generalizes “make”

For the past few weeks I’ve been working on a new tool called goals which generalizes make. I did a quick talk at Red Hat about this tool which you can download from the link below:

Video link (MP4 format, 31 minutes, 217 MB)

There are also my written notes from the talk here.

Pyrit by Řrřola, incredible raytracing demo as a qemu bootable disk image

One of the things I showed at KVM Forum last month was a cool demo by Jan Kadlec (Řrřola). Originally this was a 256 byte MSDOS COM file. I adapted it very slightly to turn it into a boot sector. Here’s how to run it using nbdkit and qemu:

nbdkit data data="
  49 192 49 219 185 255 0 191 254 255 137 252 190 0 1 189 28 9 79 176 
  19 79 208 233 205 16 15 190 203 48 205 136 233 137 200 247 224 209 
  233 254 195 120 2 134 206 184 16 16 117 228 184 79 176 163 0 1 184 19 
  79 163 2 1 184 208 233 163 4 1 184 205 16 163 6 1 184 15 190 163 8 1 
  184 203 48 163 10 1 184 205 136 163 12 1 184 233 137 163 14 1 184 49 
  71 186 202 159 142 194 96 185 12 0 1 245 96 217 69 254 217 251 217 
  238 132 193 117 2 217 224 221 219 226 246 221 219 217 193 217 69 254 
  217 251 222 204 222 201 4 127 112 241 222 195 222 233 114 233 217 26 
  41 254 123 250 97 226 204 97 66 170 96 219 227 140 195 191 252 255 
  223 6 68 125 221 23 223 69 251 223 69 252 232 14 0 97 129 195 205 204 
  115 225 117 222 228 96 72 224 152 145 0 246 112 78 0 210 112 74 185 
  12 0 1 245 217 236 216 2 86 217 2 216 204 41 254 123 248 94 222 193 
  222 193 83 217 19 133 99 2 120 2 41 251 217 192 216 15 223 242 114 6 
  216 249 217 23 137 40 222 217 91 139 87 6 59 87 2 126 16 226 199 139 
  24 217 1 216 8 216 192 216 235 41 254 123 244 217 192 222 14 70 125 
  219 29 102 193 61 22 120 24 222 60 220 201 216 202 219 27 42 67 1 219 
  27 50 67 1 36 72 4 80 246 37 136 37 195 127 112 97 66 68 78 
  @0x1fe 85 170 
  " size=512 --run 'qemu-system-x86_64 -hda $nbd'

(I would normally put a screenshot here, but it doesn’t do it justice. I suggest really running that command and also reading the surprisingly clean source code)

nbdkit new eval plugin and ip filter

nbdkit is our flexible toolkit for building block devices. I just added a couple of new features which will appear in the next stable release, nbdkit 1.18.

Previously I’ve talked on this blog and gave a talk at FOSDEM about how you can write block devices in shell script using nbdkit-sh-plugin. But that requires you to use an extra file for the script. What if opening an extra file is too much work? Well now you can specify the script directly on the nbdkit command line using the new eval plugin.

You can write code like:

nbdkit eval \
       config='ln -sf "$(realpath "$3")" $tmpdir/file' \
       get_size='stat -Lc %s $tmpdir/file' \
       pread='dd if=$tmpdir/file skip=$4 count=$3 iflag=count_bytes,skip_bytes' \
       pwrite='dd of=$tmpdir/file seek=$4 conv=notrunc oflag=seek_bytes' \
       file=disk.img

which is a complete NBD server / block device backed by a local file. Of course it’s probably easier to use nbdkit-file-plugin for this, but the shell script gives you more control like letting you simulate failures or delays.

The other new feature is connected to a CVE we had earlier this year. CVE-2019-14850 happened because nbdkit used to open the plugin as soon as any client established a TCP connection. For some plugins opening them is quite a heavyweight action (eg. it might mean that the plugin has to establish a connection to a second server). This is before NBD negotiation or TLS had started, and it allowed clients potentially to overwhelm the server with requests even if those clients would not be authorized to connect.

To fix this we delay opening plugins until after the NBD handshake (and thus TLS authentication) has completed. But this in turn meant there was no way for plugins to reject connections early, for example based on IP address. So now I have added a preconnect method which gets runs on first TCP connection and can be used to do lightweight early filtering. There is a new nbdkit-ip-filter which implements simple TCP-wrappers-style allow/deny lists.

Short talk about NBD from the KVM Forum 2019

Here’s our short talk about Network Block Device (NBD) given at the KVM Forum last month:

NBD over AF_VSOCK

How do you talk to a virtual machine from the host? How does the virtual machine talk to the host? In one sense the answer is obvious: virtual machines should be thought of just like regular machines so you use the network. However the connection between host and guest is a bit more special. Suppose you want to pass a host directory up to the guest? You could use NFS, but that’s sucky to set up and you’ll have to fiddle around with firewalls and ports. Suppose you run a guest agent reporting stats back to the hypervisor. How do they talk? Network, sure, but again that requires an extra network interface and the guest has to explicitly set up firewall rules.

A few years ago my colleague Stefan Hajnoczi ported VMware’s vsock to qemu. It’s a pure guest⟷host (and guest⟷guest) sockets API. It doesn’t use regular networks so no firewall issues or guest network configuration to worry about.

You can run NFS over vsock [PDF] if you want.

And now you can of course run NBD over vsock. nbdkit supports it, and libnbd is (currently the only!) client.