Tag Archives: tmpfs

New nbdkit “remote tmpfs” (tmpdisk plugin)

I was making some thin clients for the Fedora RISC-V project a few weeks ago. These are based on the HiFive Unleashed U540 board and so they have no local SATA, only slow, unreliable SD cards. Any filesystems that might be heavily used must be network filesystems.

As these are Linux clients we essentially have three possible choices for network filesystems: NFS, NBD or a cluster FS. As I didn’t want to set up multiple server nodes or need the redundancy, cluster filesystems are immediately discounted. NFS is — “fine”. And indeed I selected that for /home where performance is not a problem. But for the clients that are build servers I selected NBD as a high-performance block-based storage. I’m using nbdkit, the high performance flexible NBD server.

However nbdkit didn’t quite do what I wanted, so I had to write a new plugin. I needed a “remote tmpfs“.

To get a new “remote tmpfs”, a fresh filesystem each time, the client does:

modprobe nbd
nbd-client server /dev/nbd0
mount /dev/nbd0 /var/scratch

Backing this is a flexible new plugin called tmpdisk. By default it creates a new disk for each connection, formats it with mkfs and serves it to the client. But it’s also scriptable, so you can substitute any command you like instead of mkfs to create your own custom disks (I recommend looking at mke2fs -d in case you need to have pre-populated scratch disks).

It’s important to note that these are scratch disks so when the client unmounts the filesystem it is completely deleted from the server. But that’s fine for temporary directories and (for my purposes) package builds.

To find out more about nbdkit, watch my video from FOSDEM 2019.

2 Comments

Filed under Uncategorized

tmpfs considered harmful

To clarify, this is about using tmpfs for /tmp being harmful. Specialized use of tmpfs in other situations is fine.

Fedora (beginning with 18) has changed /tmp so it is now a separate tmpfs partition. I’ve asked FESCO to consider reverting this.

tmpfs is usually a bad idea.

Computer storage is hierarchical for performance reasons: CPU registers are ultra-fast but very few in number. Cache is faster than main memory. Disk drives are slow, but huge and persistent.

These are unavoidable facts, caused by the design of electronics. Every time we move down a step in this hierarchy, it introduces complexity. One of the hardest parts of writing a compiler is register allocation. Complex NP-complete algorithms have to be written, and only the most expert programmers and theorists tackle this. (For JIT allocation it’s still very much an open research area). Countless thousands of hours have been spent by kernel developers to optimize cache performance. When we have to decide what to keep in memory and what to spill to disk, complex paging algorithms are designed and tuned ad nauseam.

tmpfs introduces an entirely artificial and unnecessary step that all programmers and users have to worry about. It’s limited to at most the available memory + swap space (although by default to half of RAM). Swap is usually stored on a separate partition that cannot expand, so you can run out of swap (and hence /tmp), still have plenty of free disk space, and your machine will crash.

Everyone must now be careful never to store a file in /tmp that might grow large, nor too many files at the same time. Every last little utility must be checked, including ones that aren’t part of your distro. Every user must be “re-educated” not to use /tmp for temporary files.

Everyone must worry about whether their files need to survive a reboot. (For example the default mutt configuration stores newly drafted emails in /tmp, so you’d better hope your machine doesn’t suffer a power failure while you’re writing. Or that you don’t have to compose an email larger than half available memory.)

All of this is quite unnecessary. We already have a memory-backed storage system that transparently spills to the filesystem. It’s called … the filesystem.

The case for performance has not been made either, but even if we assume that tmpfs is vastly better than the filesystem, it’s better to fix the filesystem to make it faster than to spend any more time on tmpfs. Fixes to the filesystem benefit everyone.

It’s good that Debian reverted this change, and Fedora should too.

52 Comments

Filed under Uncategorized