Ridiculously big “files”

In the last post I showed how you can combine nbdfuse with nbdkit’s RAM disk to mount a RAM disk as a local file. In a talk I gave at FOSDEM last year I described creating these absurdly large RAM-backed filesystems and you can do the same thing now to create ridiculously big “files”. Here’s a 7 exabyte file:

$ touch /var/tmp/disk.img
$ nbdfuse /var/tmp/disk.img --command nbdkit -s memory 7E &
$ ll /var/tmp/disk.img 
 -rw-rw-rw-. 1 rjones rjones 8070450532247928832 Nov  4 13:37 /var/tmp/disk.img
$ ls -lh /var/tmp/disk.img 
 -rw-rw-rw-. 1 rjones rjones 7.0E Nov  4 13:37 /var/tmp/disk.img

What can you actually do with this file, and more importantly does anything break? As in the talk, creating a Btrfs filesystem boringly just works. mkfs.ext4 spins using 100% of CPU. I let it go for 15 minutes but it seemed no closer to either succeeding or crashing. Emacs said:

File disk.img is large (7 EiB), really open? (y)es or (n)o or (l)iterally

and I was too chicken to find out what it would do if I really opened it.

I do wonder if there’s a DoS attack here if I leave this seemingly massive regular file lying around in a public directory.

nbdkit for loopback pt 5: 8 exabyte btrfs filesystem

Thanks Chris Murphy for noting that btrfs can create and mount 8 EB (approx 2⁶³ byte) filesystems effortlessly:

$ nbdkit -fv memory size=$(( 2**63-1 ))

# modprobe nbd
# nbd-client -b 512 localhost /dev/nbd0
# blockdev --getss /dev/nbd0
512
# gdisk /dev/nbd0
Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048  18014398509481948   8.0 EiB     8300  Linux filesystem
# mkfs.btrfs -K /dev/nbd0p1
btrfs-progs v4.16
See http://btrfs.wiki.kernel.org for more information.

Detected a SSD, turning off metadata duplication.  Mkfs with -m dup if you want to force metadata duplication.
Label:              (null)
UUID:               770e5592-9055-4551-8416-b6376802a2ad
Node size:          16384
Sector size:        4096
Filesystem size:    8.00EiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         single            8.00MiB
  System:           single            4.00MiB
SSD detected:       yes
Incompat features:  extref, skinny-metadata
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     8.00EiB  /dev/nbd0p1

# mount /dev/nbd0p1 /tmp/mnt
# df -h /tmp/mnt
Filesystem      Size  Used Avail Use% Mounted on
/dev/nbd0p1     8.0E   17M  8.0E   1% /tmp/mnt

I created a few files in there and it all seemed to work although I didn’t do any extensive testing. Good job btrfs!

virt-resize now works with btrfs

virt-resize now works with guests that use btrfs:

$ virt-resize /dev/vg_pin/F15BTRFSx64 /dev/vg_pin/TmpBTRFS --expand sda3
Examining /dev/vg_pin/F15BTRFSx64 ...
[########################################################################]
**********

Summary of changes:

/dev/sda1: This partition will be left alone.

/dev/sda2: This partition will be left alone.

/dev/sda3: This partition will be resized from 9.0G to 14.0G.  The 
    filesystem btrfs on /dev/sda3 will be expanded using the 
    'btrfs-filesystem-resize' method.

**********
Setting up initial partition table on /dev/vg_pin/TmpBTRFS ...
Copying /dev/sda1 ...
[########################################################################]
Copying /dev/sda2 ...
[########################################################################]
Copying /dev/sda3 ...
[########################################################################]
[########################################################################]
Expanding /dev/sda3 using the 'btrfs-filesystem-resize' method ...

Resize operation completed with no errors.  Before deleting the old 
disk, carefully check that the resized disk boots and works correctly.

mkfs compared on different filesystems

How long does it take to mkfs a 10GB disk with all the different filesystems out there?

See my test results here using the new guestfish sparse / filesystem support. btrfs is “best” and ext3 comes off “worst”.

As a test this is interesting, but it’s not that relevant for most users — they will be most interested in how well the filesystem performs for their workload, which is not affected by mkfs time and hard to measure in general benchmarks anyway.

Update

In response to Stephen’s comment, I retested this using a memory-backed block device so there is no question about whether the host backing store affects the test:

$ for fs in ext2 ext3 ext4 xfs jfs reiserfs nilfs2 ntfs msdos btrfs hfs hfsplus gfs gfs2
    do guestfish sparse /dev/shm/test.img 10G : run : echo $fs : sfdiskM /dev/sda , : \
        time mkfs $fs /dev/sda1
    done
ext2
elapsed time: 1.45 seconds
ext3
elapsed time: 2.71 seconds
ext4
elapsed time: 2.58 seconds
xfs
elapsed time: 0.13 seconds
jfs
elapsed time: 0.27 seconds
reiserfs
elapsed time: 0.33 seconds
nilfs2
elapsed time: 0.08 seconds
ntfs
elapsed time: 2.07 seconds
msdos
elapsed time: 0.14 seconds
btrfs
elapsed time: 0.07 seconds
hfs
elapsed time: 0.17 seconds
hfsplus
elapsed time: 0.17 seconds
gfs
elapsed time: 0.84 seconds
gfs2
elapsed time: 2.76 seconds

Filesystem metadata overhead

Which filesystems have the largest metadata overhead[1]? We can find out using guestfish sparse file support and a short shell script:

#!/bin/sh -

guestfish=fish/guestfish
testimg=/mnt/tmp/test/test.img
size=1G

for fstype in ext2 ext3 ext4 xfs ntfs msdos ; do
    $guestfish <<EOF
      sparse $testimg $size
      run
      sfdiskM /dev/sda ,
      mkfs $fstype /dev/sda1
EOF
    echo -n "$fstype: "
    du $testimg
done

The results:

Filesystem	Allocated kilobytes (out of 1G)	Overhead %
ext2	16948	1.6%
ext3 [2]	33352	3.2%
ext4 [2]	33288	3.2%
xfs	5132	0.5%
ntfs [3]	5748	0.5%
msdos & vfat	2076	0.2%
reiserfs [3]	32916	3.1%
btrfs [3]	4224	0.4%
hfs & hfsplus [3]	16432	1.6%
nilfs2 [3]	2060	0.2%
jfs [3]	4364	0.4%
gfs [3]	16612	1.6%
gfs2 [3,4]	132576	12%

Notes

[1] Yes, there are shortcomings in the methodology. What this really measures is how many blocks are written by the mkfs program. Even if you believe the figures, this only measures the initial overhead, but you can use the same technique to measure the overhead of storing (eg) lots of small files, or whatever is appropriate for your workload.
[2] Note the difference between ext2 and ext3/4 seems to be entirely down to the size of the journal, which is a kind of metadata overhead, but one that you can easily control.
[3] Needs three small patches to libguestfs to get these filesystems to work.
[4] This seems off the scale — needs further investigation.