New in libguestfs: Allow cache mode to be selected

libguestfs used to default to the qemu cache=none caching mode. By allowing other qemu caching modes to be used, you can get a considerable speed up, up to 25% in the best case.

qemu has a pretty confusing range of caching modes. It’s possible to select writeback (the default), writethrough, none, directsync or unsafe. Know what all those mean? Congratulations, you are Kevin Wolf.

It helps to understand there are many places where your data could be cached. Some are: in the guest’s RAM, in the host’s RAM, or in the small cache which is part of the circuitry on the disk drive. And there’s a conflict of interests too. For your precious novel, you’d hope that when you hit “Save” (and even before) it has actually been written in magnetic patterns on the spinning disk. But the vast majority of data that is written (eg. during a large compile) is not that important, can easily be reconstructed, and is best left in RAM for speedy access.

Qemu emulates disk drives that in some way resemble the real thing, and disk drives have a range of ways that the operating system tells the drive the importance of data and when it needs to be persisted. For example a guest using a (real or emulated) SCSI drive sets the Force Unit Access bit to force the data to be written to physical media before being acknowledged. Unfortunately not all guests know about the FUA flag or its equivalents, and even those that do sometimes don’t issue these flush commands in the right place. For example in very old RHEL, filesystems would issue the flush commands correctly, but if the filesystem used LVM underneath, LVM would not pass these commands through to the hardware.

So now to the qemu cache modes:

• writeback lets writes be cached in the host cache. This is only safe provided the guest issues flush commands correctly (which translate into fdatasync on the host). If your guest is old and/or doesn’t do this, then you need:
• writethrough is the same as writeback, but all writes are flushed to disk as well because qemu can never know when a write is important. This is of course dead slow.
• none uses O_DIRECT on the host side, avoiding the host cache at all. This is not especially useful, and in particular it means you’re not using the potentially large host RAM to cache things.
• directsync is like none + writethrough (and like writethrough it’s only applicable if your guest doesn’t send flush commands properly).
• Finally unsafe is the bad boy of caching modes. Flush commands are ignored. That’s fast, but your data is toast if the machine crashes, even if you thought you did a sync.

Libguestfs ≥ 1.23.20 does not offer all this choice. For a start, libguestfs always uses a brand new kernel, so we can assume that flush commands work, and that means we can ignore writethrough and directsync.

none (ie. O_DIRECT) which is what libguestfs has always used up to now has proven to be an endless source of pain particularly for people who are already suffering under tmpfs. It also has the big disadvantage of bypassing the host RAM for caching.

That leaves writeback and unsafe which are the two choices of cachemode that you now have when using the libguestfs add_drive API. writeback is the new, safe default. Flush commands are obeyed so as long as you’re using a journalled filesystem or issue guestfs_sync calls your data will be safe. And there’s a small performance benefit because we are using the host RAM to cache writes.

cachemode=unsafe is the dangerous new choice you have. For scratch disks, testing, temporary disks, basically anything where you either don’t care about the data, or can easily reconstruct the data, this will give you a nice performance boost (I measured about 25%).