Look:
$ nbdkit -f -v memory size=$(( 2**63-1 ))
On the same machine:
# modprobe nbd # nbd-client localhost /dev/nbd0 Warning: the oldstyle protocol is no longer supported. This method now uses the newstyle protocol with a default export Negotiation: ..size = 8796093022207MB Connected /dev/nbd0 # sgdisk -n 1 /dev/nbd0 Creating new GPT entries in memory. The operation has completed successfully. # gdisk -l /dev/nbd0 Number Start (sector) End (sector) Size Code Name 1 1024 9007199254740973 8.0 EiB 8300
What’s happening here is we’re creating an 8 exabyte disk backed by an nbdkit instance which is storing the data in a sparse array in memory.
You can then try fun things like creating massive XFS filesystems.
A few caveats:
- You must be using kernel ≥ 4.18. Earlier kernels had a bunch of bugs in the Linux NBD driver.
- You must apply this fix for this bug in the nbd-client program. I have updated the Fedora packages so make sure you get the latest update from Fedora updates-testing.
- You will need nbdkit ≥ 1.6 which has support for sparse memory-backed huge disks.
I think it’d be interesting to integrate this into filesystem test suites. Unfortunately use of the Linux NBD kernel driver needs root 😦
Richard WM Jones 
Should this work on Fedora 28 which has nbdkit-1.4.3-1.fc28, and nbd-3.17-3.fc28 and kernel-4.18.5-300.fc29.x86_64? Because I get
$ sudo nbdkit -f -v memory size=$(( 2**63-1 ))
nbdkit: debug: TLS disabled: could not load TLS certificates
nbdkit: debug: registering plugin /usr/lib64/nbdkit/plugins/nbdkit-memory-plugin.so
nbdkit: debug: registered plugin /usr/lib64/nbdkit/plugins/nbdkit-memory-plugin.so (name memory)
nbdkit: debug: memory: load
nbdkit: debug: memory: config key=size, value=9223372036854775807
nbdkit: debug: memory: config_complete
nbdkit: error: cannot allocate disk: Cannot allocate memory
You need nbdkit ≥ 1.6 which has huge sparse array support. The version you’re using (1.4.3) is trying to allocate 8 exabytes of RAM in a flat array and that ain’t gonna work for a lot of different reasons.
Edit: I guess in Fedora 28 you can probably install the F29 package:
OK it wanted to install 1.5.10, looks like the testing request is still pending for 1.6.0. But pulling nbdkit-bash-completion nbdkit-basic-plugins nbdkit nbdkit-server nbdkit-basic-filters directly from koji does install and works on Fedora 28.
mkfs.btrfs -K on this 8EiB device took about 1 second, so did mounting it. Did a quick test copy of a file, scrub, and btrfs check. So at least btrfs-progs 4.17.1 and kernel 4.18.5 aren’t seemingly confused by the 1K logical/physical sector size that’s being reported, or anything else.
For fs regression testing I think a configurable sector size would be ideal, what are the chances of that happening at some future point?
On a separate but related note, when I do Fedora installs, I see in top that the loop0 device process is soaking the CPU 70-100% for the duration of the installation; and also the installation seems a lot slower than I’d guess it should be (i.e. the device reads from the source containing the loop0’s backing file are not anywhere near the typical file system read speeds I see from that same device). So yeah I don’t know why that is but any improvements for loopback mounts performance/resource wise I suspect would be very welcome!
Actually it turned out that you can already set the sector size using the
nbd-client -boption.> You must be using kernel ≥ 4.18. Earlier kernels had a bunch of bugs in the Linux NBD driver.
This is just wonderful to hear, as someone who actively uses NBD on the 4.14 series (and not going to upgrade yet). Could you link to what are the bugs (e.g. bko entries)? Of course I can’t tell you what to write on your blog, but surely you understand how this line looks in this context.
They were all fixed by Josef Bacik. Google for “josef bacik nbd bugs”.
Pingback: nbdkit for loopback pt 5: 8 exabyte btrfs filesystem | Richard WM Jones
Pingback: nbdkit for loopback pt 6: giant file-backed disks for testing | Richard WM Jones