nbdkit / FOSDEM test presentation about better loop mounts for Linux

I’ve submitted a talk about nbdkit, our flexible pluggable NBD server, to FOSDEM next February. This is going to be about using NBD as a better way to do loop mounts in Linux.

In preparation I gave a very early version of the talk to a small Red Hat audience.

Video link: http://oirase.annexia.org/rwmj.wp.com/rjones-nbdkit-tech-talk-2018-11-19.mp4

Sorry about the slow start. You may want to skip to 2 mins to get past the intro.

Summary of what’s in the talk:

1. Demo of regular, plain loop mounting.
2. Demo of loop mounting an XZ-compressed disk image using NBD + nbdkit.
3. Slides about how loop device compares to NBD.
4. Slides about nbdkit plugins and filters.
5. Using VMware VDDK to access a VMDK file.
6. Creating a giant disk costing EUR 300 million(!)
7. Visualizing a single filesystem.
8. Visualizing RAID 5.
9. Writing a plugin in shell script (live demo).
10. Summary.

New in nbdkit: Create an ISO image on the fly

nbdkit is the pluggable Network Block Device server that Eric and I wrote. I have submitted a talk to FOSDEM next February about the many weird and wonderful ways you can use nbdkit as a flexible replacement for loopback mounting.

Anyway, new in nbdkit 1.7.6 you can now create ISO 9660 (CD-ROM) disk images on the fly from a directory:

# nbdkit iso /boot params="-JrT"
# nbd-client -b 512 localhost /dev/nbd0
# file -bsL /dev/nbd0
ISO 9660 CD-ROM filesystem data 'CDROM'
# mount /dev/nbd0 /tmp/mnt
# ls /tmp/mnt
config-4.18.0-0.rc8.git2.1.fc29.x86_64
config-4.19.0-0.rc1.git3.2.fc30.x86_64
config-4.19.0-0.rc6.git0.1.fc30.x86_64
efi
extlinux
grub2
[etc]
# umount /tmp/mnt
# nbd-client -d /dev/nbd0
# killall nbdkit

That ISO wouldn’t actually be bootable, but you could create one (eg. an El Torito ISO) by adding the appropriate extra parameters.

To head off the first question: If you copy files into the directory while nbdkit is running, do they appear in the ISO? Answer: No! This is largely impossible with the way Linux block devices work.

nbdkit for loopback pt 5: 8 exabyte btrfs filesystem

Thanks Chris Murphy for noting that btrfs can create and mount 8 EB (approx 2⁶³ byte) filesystems effortlessly:

$ nbdkit -fv memory size=$(( 2**63-1 ))

# modprobe nbd
# nbd-client -b 512 localhost /dev/nbd0
# blockdev --getss /dev/nbd0
512
# gdisk /dev/nbd0
Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048  18014398509481948   8.0 EiB     8300  Linux filesystem
# mkfs.btrfs -K /dev/nbd0p1
btrfs-progs v4.16
See http://btrfs.wiki.kernel.org for more information.

Detected a SSD, turning off metadata duplication.  Mkfs with -m dup if you want to force metadata duplication.
Label:              (null)
UUID:               770e5592-9055-4551-8416-b6376802a2ad
Node size:          16384
Sector size:        4096
Filesystem size:    8.00EiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         single            8.00MiB
  System:           single            4.00MiB
SSD detected:       yes
Incompat features:  extref, skinny-metadata
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     8.00EiB  /dev/nbd0p1

# mount /dev/nbd0p1 /tmp/mnt
# df -h /tmp/mnt
Filesystem      Size  Used Avail Use% Mounted on
/dev/nbd0p1     8.0E   17M  8.0E   1% /tmp/mnt

I created a few files in there and it all seemed to work although I didn’t do any extensive testing. Good job btrfs!

nbdkit for loopback pt 4: loopback-mounting compressed images

nbdkit is a pluggable NBD server with lots of plugins and filters. Two of the plugins[1] handle compressed files (for gzip and xz respectively). We can uncompress and serve a file on the fly. For gzip it’s kind of inefficient. For xz it’s very efficient provided you prepared your xz files ahead of time with a smaller than default block size.

Let’s use nbdkit to loopback mount an xz file:

$ nbdkit -fv xz file=/var/tmp/fedora-28.img.xz

# nbd-client -b 512 localhost /dev/nbd0
Warning: the oldstyle protocol is no longer supported.
This method now uses the newstyle protocol with a default export
Negotiation: ..size = 6144MB
Connected /dev/nbd0
# ls /dev/nbd0p*
/dev/nbd0p1  /dev/nbd0p2  /dev/nbd0p3  /dev/nbd0p4
# fdisk -l /dev/nbd0
Device        Start      End Sectors  Size Type
/dev/nbd0p1    2048     4095    2048    1M BIOS boot
/dev/nbd0p2    4096  2101247 2097152    1G Linux filesystem
/dev/nbd0p3 2101248  3360767 1259520  615M Linux swap
/dev/nbd0p4 3360768 12580863 9220096  4.4G Linux filesystem
# mount -o ro /dev/nbd0p4 /mnt

Of course it’s read-only. To write to a compressed file would involve changing the size of inner parts of the file. Use qcow2 compression if you want a writable compressed file (although writes to that format are not compressed).

Also loopback mounting in general is unsafe. Use libguestfs to safely mount untrusted disk images.

[1] These should really be filters, not plugins, so that you can chain an uncompression filter into an existing plugin, and one day I’ll get around to writing that.

nbdkit for loopback pt 3: loopback mounting VMware disks

nbdkit is a pluggable NBD server and it comes with a very wide range of plugins (of course you can also write your own). One of them is the VMware VDDK plugin, an interface between nbdkit and the very proprietary VMware VDDK library. The library allows you to read local VMware disks or access remote VMware servers. In this example I’m going to use it to loopback mount a VMDK file:

$ export LD_LIBRARY_PATH=~/tmp/vddk/vmware-vix-disklib-distrib/lib64
$ nbdkit -fv vddk \
    libdir=~/tmp/vddk/vmware-vix-disklib-distrib \
    file=/var/tmp/fedora.17.x86-64.20120529.vmdk

When loopback-mounting you must use a 512 byte sector size (see the mailing list for discussion):

# nbd-client -b 512 localhost /dev/nbd0
Warning: the oldstyle protocol is no longer supported.
This method now uses the newstyle protocol with a default export
Negotiation: ..size = 10240MB
Connected /dev/nbd0

Standard health warning: Loopback mounting any unknown disk is dangerous! You should use libguestfs instead as it protects you from harmful disks and also doesn’t require root.

It turns out this VMDK file contains a partitioned disk with one partition:

# fdisk -l /dev/nbd0
Disk /dev/nbd0: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 1024 bytes / 1024 bytes
Disklabel type: dos
Disk identifier: 0x000127ae

Device      Boot Start      End  Sectors Size Id Type
/dev/nbd0p1       2048 20971519 20969472  10G 83 Linux

and it can be mounted directly and it’s fully writable:

# mount /dev/nbd0p1 /mnt
# ls -l /mnt
total 76
lrwxrwxrwx.  1 root root     7 May 29  2012 bin -> usr/bin
dr-xr-xr-x.  4 root root  4096 May 29  2012 boot
drwxr-xr-x.  2 root root  4096 Feb  3  2012 dev
drwxr-xr-x. 59 root root  4096 May 29  2012 etc
drwxr-xr-x.  2 root root  4096 Feb  3  2012 home
lrwxrwxrwx.  1 root root     7 May 29  2012 lib -> usr/lib
lrwxrwxrwx.  1 root root     9 May 29  2012 lib64 -> usr/lib64
[etc]
# touch /mnt/hello