Does virt-v2v preserve sparseness?

A common question is does virt-v2v preserve sparseness (aka thin provisioning) when you convert the guest from VMware to a KVM-based hypervisor like oVirt/RHV or OpenStack? The very short answer is: “what?”. The medium answer is: The question doesn’t really make sense. For the long answer, read on …

First of all we need to ask what is thin provisioning? For complex and essentially historical reasons when you provision a new guest you have to decide how much maximum disk space you want it to have. Let’s say you choose 10 GB. However because a new guest install might only take, say, 1 GB, you can also decide if you want the whole 10 GB to be preallocated up front or if you want the disk to be thin provisioned or sparse, meaning it’ll only take up 1 GB of host space, but that will increase as the guest creates new files. There are pros and cons to preallocation. Preallocating the full 10 GB obviously takes a lot of extra space (and what if the guest never uses it?), but it can improve guest performance in some circumstances.

That is what happens initially. By the time we come to do a virt-v2v conversion that guest may have been running for years and years. Now what does the disk look like? It doesn’t matter if the disk was initially thin provisioned or fully allocated, what matters is what the guest did during those years.

Did it repeatedly fill up the disk and/or delete those files? – In which case your initially thin provisioned guest could now be fully allocated.

Did it have trimming enabled? Your initially preallocated guest might now have become sparsely allocated.

In any case VMware doesn’t store this initial state, nor does it make it very easy to find out which bits of the disk are actually backed by host storage and which bits are sparse (well, maybe this is possible, but not using the APIs we use when liberating your guests from VMware).

In any case, as I explained in this talk (slides) from a few years ago, virt-v2v tries to avoid copying any unused, zeroed or deleted parts of the disk for efficiency reasons, and so it will always make the disk maximally sparse when copying it (subject to what the target hypervisor does, read on).

When virt-v2v comes to creating the target guest, the default is to create a maximally sparse guest, but there are two ways to change this:

1. You can specify the -oa preallocated option, where virt-v2v will try to ask the target hypervisor to fully preallocate the target disks of the guest.
2. For some hypervisors, especially RHV, your choice of backend storage may mean you have no option but to use preallocated disks (unfortunately I cannot give clear advice here, best to ask a RHV expert).

The basic rule is that when converting guests you need to think about whether you want the guest to be sparse or preallocated after conversion, based on your own performance vs storage criteria. Whether it happened to be thin provisioned when you set it up on VMware years earlier isn’t a relevant issue.

Filesystem metadata overhead

Which filesystems have the largest metadata overhead[1]? We can find out using guestfish sparse file support and a short shell script:

#!/bin/sh -

guestfish=fish/guestfish
testimg=/mnt/tmp/test/test.img
size=1G

for fstype in ext2 ext3 ext4 xfs ntfs msdos ; do
    $guestfish <<EOF
      sparse $testimg $size
      run
      sfdiskM /dev/sda ,
      mkfs $fstype /dev/sda1
EOF
    echo -n "$fstype: "
    du $testimg
done

The results:

Filesystem	Allocated kilobytes (out of 1G)	Overhead %
ext2	16948	1.6%
ext3 [2]	33352	3.2%
ext4 [2]	33288	3.2%
xfs	5132	0.5%
ntfs [3]	5748	0.5%
msdos & vfat	2076	0.2%
reiserfs [3]	32916	3.1%
btrfs [3]	4224	0.4%
hfs & hfsplus [3]	16432	1.6%
nilfs2 [3]	2060	0.2%
jfs [3]	4364	0.4%
gfs [3]	16612	1.6%
gfs2 [3,4]	132576	12%

Notes

[1] Yes, there are shortcomings in the methodology. What this really measures is how many blocks are written by the mkfs program. Even if you believe the figures, this only measures the initial overhead, but you can use the same technique to measure the overhead of storing (eg) lots of small files, or whatever is appropriate for your workload.
[2] Note the difference between ext2 and ext3/4 seems to be entirely down to the size of the journal, which is a kind of metadata overhead, but one that you can easily control.
[3] Needs three small patches to libguestfs to get these filesystems to work.
[4] This seems off the scale — needs further investigation.

Terabyte virtual disks

This is fun. I added a new command to guestfish which lets you create sparse disk files. This makes it really easy to test out the limits of partitions and Linux filesystems.

Starting modestly, I tried a 1 terabyte disk:

$ guestfish

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help with commands
      'quit' to quit the shell

><fs> sparse /tmp/test.img 1T
><fs> run

The real disk image so far isn’t so big, just 4K according to “du”:

$ ll -h /tmp/test.img 
-rw-rw-r-- 1 rjones rjones 1T 2009-11-04 17:52 /tmp/test.img
$ du -h /tmp/test.img
4.0K	/tmp/test.img

Let’s partition it:

><fs> sfdiskM /dev/vda ,

The partition table only uses 1 sector, so the disk image has increased to just 8K. Let’s make an ext2 filesystem on the first partition:

><fs> mkfs ext2 /dev/vda1

This command takes some time, and the sparse disk file has grown. To 17 GB, so ext2 has an approx 1.7% overhead.

We can mount the filesystem and look at it:

><fs> mount /dev/vda1 /
><fs> df-h 
Filesystem            Size  Used Avail Use% Mounted on
/dev/vda1            1008G   72M  957G   1% /sysroot

Can we try this with larger and larger virtual disks? In theory yes, in practice the 1.7% overhead proves to be a problem. A 10T experiment would require a very real 170GB of local disk space, and where I was hoping to go, 100T and beyond, would be too large for my test machines.

In fact there is another limitation before we reach there. Local sparse files on my host ext4 filesystem are themselves limited to under 16T:

><fs> sparse /tmp/test.img 16T
write: File too large
><fs> sparse /tmp/test.img 15T

Although the appliance does boot with that 15T virtual disk:

><fs> blockdev-getsize64 /dev/vda 
16492674416640

Update

I noticed from Wikipedia that XFS has a maximum file size of 8 exabytes – 1 byte. By creating a temporary XFS filesystem on the host, I was able to create a 256TB virtual disk:

><fs> sparse /mnt/tmp/test/test.img 256T
><fs> run
><fs> blockdev-getsize64 /dev/vda 
281474976710656

Unfortunately at this point things break down. MBR partitions won’t work on such a huge disk, or at least sfdisk can’t partition it correctly.

I’m not sure what my options are at this point, but at least this is an interesting experiment in hitting limitations.