Build a guest library

I have this exported over NFS to all my local machines:

$ ls -lh /mnt/media/guest-library/
total 12G
-rw-r--r--. 1 rjones rjones 971M Apr 16 23:38 debian5x64.img.xz
-rw-r--r--. 1 rjones rjones 250M Jul 26 23:09 f16x32.img.xz
-rw-r--r--. 1 rjones rjones 809M Jul 26 20:29 f17x64.img.xz
-rw-r--r--. 1 rjones rjones  59M Jul  6 22:19 freedos11.img.xz
-rw-r--r--. 1 rjones rjones 379M Jul  7 20:11 opensuse1113x64.img.xz
-rw-r--r--. 1 rjones rjones 349M Jul  6 22:31 plan9.img.xz
-rw-rw-r--. 1 rjones rjones  126 Apr 16 17:22 README
-rw-r--r--. 1 rjones rjones 384M Apr 18 00:19 rhel3x64.img.xz
-rw-r--r--. 1 rjones rjones 471M Apr 18 00:15 rhel4x64.img.xz
-rw-r--r--. 1 rjones rjones 848M Apr 16 17:02 rhel5epelx64.img.xz
-rw-r--r--. 1 rjones rjones 768M Jul  7 07:21 sl60x64.img.xz
-rw-r--r--. 1 rjones rjones 148M May 23 16:08 ubuntu1204ppc.img.xz
-rw-r--r--. 1 rjones rjones 1.1G Apr 16 17:09 win2003.img.xz
-rw-r--r--. 1 rjones rjones 3.4G Jun  1 10:46 win8-preview.img.xz
-rw-r--r--. 1 rjones rjones 2.1G Apr 16 20:28 winxp.img.xz

Having a library of images which are (relatively) small like this is great for testing. If someone gives me a bug and it happens to affect one of the guests I have saved, then I can just unpack that guest, virt-resize it to the right size and off I go, and that’s usually a lot quicker than installing a new guest, particularly when you’ve got only slow ADSL.

Adding a new guest is easy using virt-sparsify and xz. Note I’ve set the temporary directory (TMPDIR) to somewhere with plenty of space.

$ sudo TMPDIR=$(pwd) virt-sparsify /dev/vg_pin/F18Rawhidex32 f18rawhidex32.img
Create overlay file to protect source disk ...
Examine source disk ...
 100% ⟦▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓⟧ 00:00
Fill free space in /dev/vda1 with zero ...
 100% ⟦▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓⟧ --:--
Fill free space in /dev/vg_f18rawhidex32/lv_root with zero ...
 100% ⟦▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓⟧ --:--
Clearing Linux swap on /dev/vg_f18rawhidex32/lv_swap ...
 100% ⟦▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓⟧ 00:00
Copy to destination and make sparse ...

Sparsify operation completed with no errors.  Before deleting the old 
disk, carefully check that the target disk boots and works correctly.

$ ll -h f18rawhidex32.img
-rw-r--r--. 1 root root 10G Jul 27 14:42 f18rawhidex32.img
$ du -sh f18rawhidex32.img
3.0G	f18rawhidex32.img
$ xz -T 0 --best f18rawhidex32.img
$ ll -h f18rawhidex32.img.xz
-rw-r--r--. 1 rjones rjones 667M Jul 27 14:42 f18rawhidex32.img.xz

Using virt-sparsify and xz, we can really compress VMs for storage

17G	debian5x64.img.orig

The original image has about 4.7 GB of data, plus a large swap partition, according to virt-df:

$ virt-df -a debian5x64.img.orig -h
Filesystem                        Size       Used  Available  Use%
debian5x64.img.orig:/dev/sda1     322M        66M       239M   21%
debian5x64.img.orig:/dev/debian5x64.home.annexia.org/home
                                  3.4G       359M       2.9G   11%
debian5x64.img.orig:/dev/debian5x64.home.annexia.org/root
                                  320M       301M       2.4M   95%
debian5x64.img.orig:/dev/debian5x64.home.annexia.org/tmp
                                  300M       8.2M       276M    3%
debian5x64.img.orig:/dev/debian5x64.home.annexia.org/usr
                                  3.4G       2.0G       1.2G   60%
debian5x64.img.orig:/dev/debian5x64.home.annexia.org/var
                                  2.6G       2.0G       536M   76%

Using virt-sparsify, all unused space in the image is made sparse. The most recent version can sparsify swap partitions too:

4.6G	debian5x64.img

xz --best -T 0 reduces the final image to under a gigabyte:

971M	debian5x64.img.xz

New in virt-sparsify

New in libguestfs 1.17.18, virt-sparsify can now find and sparsify even more space in your guests — Linux swap partitions are detected and zeroed (though not if they contain hibernation data).

The --zero option can be used to remove unwanted partitions.

The new version is also faster and uses less temporary space during the first stage.

New tool: virt-sysprep

New in libguestfs 1.13.19 is a tool called virt-sysprep which makes it easier to clone virtual machines.

The new tool isn’t a complete clone tool, but you can easily combine it with other tools like dd, the old virt-clone, virt-resize and virt-sparsify to clone virtual machines.

New tool for making VMs sparse (aka thin provisioning)

New in libguestfs 1.13.17 is a tool for making virtual machines or disk images sparse, also known as thin provisioning.

This tool restores sparseness lost after you’ve created and deleted files. It works for virtually any guest OS, and supports almost any filesystem.

It can also do some clever things like finding sparseness in the unused parts of LVM physical volumes, or converting between formats (eg. raw to thin-provisioned qcow2).

It’s also (like virt-resize) a temporary tool: hopefully one day KVM and guests will be able to resize and trim themselves. That day isn’t here yet.

virt-sparsify manual page