nbdkit as a flexible alternative to loopback mounts


$ nbdkit -f -v memory size=$(( 2**63-1 ))

On the same machine:

# modprobe nbd
# nbd-client localhost /dev/nbd0
Warning: the oldstyle protocol is no longer supported.
This method now uses the newstyle protocol with a default export
Negotiation: ..size = 8796093022207MB
Connected /dev/nbd0
# sgdisk -n 1 /dev/nbd0
Creating new GPT entries in memory.
The operation has completed successfully.
# gdisk -l /dev/nbd0
Number  Start (sector)    End (sector)  Size       Code  Name
   1            1024  9007199254740973   8.0 EiB     8300  

What’s happening here is we’re creating an 8 exabyte disk backed by an nbdkit instance which is storing the data in a sparse array in memory.

You can then try fun things like creating massive XFS filesystems.

A few caveats:

I think it’d be interesting to integrate this into filesystem test suites. Unfortunately use of the Linux NBD kernel driver needs root 😦


Filed under Uncategorized

8 responses to “nbdkit as a flexible alternative to loopback mounts

  1. Chris Murphy

    Should this work on Fedora 28 which has nbdkit-1.4.3-1.fc28, and nbd-3.17-3.fc28 and kernel-4.18.5-300.fc29.x86_64? Because I get
    $ sudo nbdkit -f -v memory size=$(( 2**63-1 ))
    nbdkit: debug: TLS disabled: could not load TLS certificates
    nbdkit: debug: registering plugin /usr/lib64/nbdkit/plugins/nbdkit-memory-plugin.so
    nbdkit: debug: registered plugin /usr/lib64/nbdkit/plugins/nbdkit-memory-plugin.so (name memory)
    nbdkit: debug: memory: load
    nbdkit: debug: memory: config key=size, value=9223372036854775807
    nbdkit: debug: memory: config_complete
    nbdkit: error: cannot allocate disk: Cannot allocate memory

    • rich

      You need nbdkit ≥ 1.6 which has huge sparse array support. The version you’re using (1.4.3) is trying to allocate 8 exabytes of RAM in a flat array and that ain’t gonna work for a lot of different reasons.

      Edit: I guess in Fedora 28 you can probably install the F29 package:

      dnf update --best nbdkit --releasever=29
      • Chris Murphy

        OK it wanted to install 1.5.10, looks like the testing request is still pending for 1.6.0. But pulling nbdkit-bash-completion nbdkit-basic-plugins nbdkit nbdkit-server nbdkit-basic-filters directly from koji does install and works on Fedora 28.

        mkfs.btrfs -K on this 8EiB device took about 1 second, so did mounting it. Did a quick test copy of a file, scrub, and btrfs check. So at least btrfs-progs 4.17.1 and kernel 4.18.5 aren’t seemingly confused by the 1K logical/physical sector size that’s being reported, or anything else.

        For fs regression testing I think a configurable sector size would be ideal, what are the chances of that happening at some future point?

        On a separate but related note, when I do Fedora installs, I see in top that the loop0 device process is soaking the CPU 70-100% for the duration of the installation; and also the installation seems a lot slower than I’d guess it should be (i.e. the device reads from the source containing the loop0’s backing file are not anywhere near the typical file system read speeds I see from that same device). So yeah I don’t know why that is but any improvements for loopback mounts performance/resource wise I suspect would be very welcome!

      • rich

        Actually it turned out that you can already set the sector size using the nbd-client -b option.

  2. romanrm

    > You must be using kernel β‰₯ 4.18. Earlier kernels had a bunch of bugs in the Linux NBD driver.

    This is just wonderful to hear, as someone who actively uses NBD on the 4.14 series (and not going to upgrade yet). Could you link to what are the bugs (e.g. bko entries)? Of course I can’t tell you what to write on your blog, but surely you understand how this line looks in this context.

  3. Pingback: nbdkit for loopback pt 5: 8 exabyte btrfs filesystem | Richard WM Jones

  4. Pingback: nbdkit for loopback pt 6: giant file-backed disks for testing | Richard WM Jones

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.