Tag Archives: sparse

Compressed RAM disks

There was a big discussion last week about whether zram swap should be the default in a future version of Fedora.

This lead me to think about the RAM disk implementation in nbdkit. In nbdkit up to 1.20 it supports giant virtual disks up to 8 exabytes using a sparse array implemented with a 2-level page table. However it’s still a RAM disk and so you can’t actually store more real data in these disks than you have available RAM (plus swap).

But what if we compressed the data? There are some fine, very fast compression libraries around nowadays — I’m using Facebook’s Zstandard — so the overhead of compression can be quite small, and this lets you make limited RAM go further.

So I implemented allocators for nbdkit ≥ 1.22, including:

$ nbdkit memory 1T allocator=zstd

Compression ratios can be really good. I tested this by creating a RAM disk and filling it with a filesystem containing text and source files, and was getting 10:1 compression. (Note that filesystems start with very regular, easily compressible metadata, so you’d expect this ratio to quickly drop if you filled the filesystem up with a lot of files).

The compression overhead is small, although the current nbdkit-memory-plugin isn’t very smart about locking so it has rather poor performance under multi-threaded loads anyway. (A fun little project to fix that for someone who loves pthread and C.)

I also implemented allocator=malloc which is a non-sparse direct-mapped RAM disk. This is simpler and a bit faster, but has rather obvious limitations compared to using the sparse allocator.

2 Comments

Filed under Uncategorized

Tip: Making a disk image sparse

Update: libguestfs ≥ 1.14 includes a new tool called virt-sparsify which can make guests sparse (thin-provisioned).

A sparse file is one where file blocks that would contain all zeroes are omitted from the file (and don’t take up any space in the filesystem). A sparse virtual disk image is the same sort of thing: blocks that the guest hasn’t written to yet are not stored by the host, and read as all zeroes. Sparse disk images can be implemented using sparse files on the host, or you can use a format like qcow2 which inherently supports sparse files.

The problem with sparse files is that they gradually grow. When a guest writes a block it is allocated, and potentially this is never freed, even if the guest deletes the file or writes all zeroes to the block. [Eventually this problem will be solved by implementing the TRIM command which lets the host know that the guest no longer requires a block, but we’re not quite there yet.]

This is of course a problem if you fill up the guest disk and then delete the files. The host file does not regain its sparseness.

How do you therefore sparsify a disk image?

There is a technique that you can use, which is simple to understand and implement, but it does require taking the guest offline.

First, fill the empty space in the guest with zeroes. A simple way to do this for a Linux guest is to run this command (run it within each guest filesystem):

dd if=/dev/zero of=zerofile bs=1M
# note that the 'dd' command fills up all free space and eventually fails
sync
rm zerofile

Now shut down the guest.

Copy the guest disk image using either qemu-img convert or cp --sparse=always. “cp” is the fastest but only works to sparsify a raw-format disk image:

cp --sparse=always guest-disk.img guest-disk-copy.img

A little-known feature of the qemu-img convert subcommand is that it automatically sparsifies any disk sector which contains all zeroes, and of course it can convert the format at the same time:

qemu-img convert -f raw -O qcow2 guest-disk.img guest-disk-copy.qcow2

Now the copy in both cases is sparsified, and hopefully a lot smaller than before.

Addendum: Instead of running “dd” by hand inside each guest, you can use the following libguestfs script to achieve the same (but note the guest must be shut down otherwise you will get disk corruption):

#!/usr/bin/perl -w
# ./phil-space.pl (disk.img|GuestName)
# Requires libguestfs >= 1.5.

use strict;
use Sys::Virt;
use Sys::Guestfs;
use Sys::Guestfs::Lib qw(open_guest);

die "$0: recent version of libguestfs >= 1.5 is required\n"
    unless defined (Sys::Guestfs->can ("list_filesystems"));

die "$0 (disk.img|GuestName)\n" unless @ARGV >= 1;

my $g = open_guest (\@ARGV, rw => 1);
$g->launch ();

my %filesystems = $g->list_filesystems ();

foreach (keys %filesystems) {
    eval {
        $g->mount_options ("", $_, "/");

        print "filling empty space in $_ with zeroes ...\n";

        my $filename = "/deleteme.tmp";
        eval { $g->dd ("/dev/zero", $filename) };
        $g->sync (); # make sure the last part of the file is written
        $g->rm ($filename);
    };
    $g->umount_all ();
}

$g->sync ()

2 Comments

Filed under Uncategorized