Tag Archives: virt-sparsify

New in virt-sparsify: In place sparsification

New in virt-sparsify ≥ 1.25.44, you can now sparsify disk images without copying them, so-called in-place sparsification.

It’s easy to use:

$ virt-sparsify --in-place fedora.img
Trimming /dev/sda1 ...
Clearing Linux swap on /dev/sda2 ...
Trimming /dev/sda3 ...

Sparsify in-place operation completed with no errors.

… and much faster. However it does require very recent kernel and qemu support.

Thanks: Paolo Bonzini, Eric Sandeen & Kevin Wolf for implementing discard support and patiently helping out when we started to test and use it.

18 Comments

Filed under Uncategorized

FOSDEM lightning talk on libguestfs next Sunday

If you’re around at FOSDEM next Sunday, please drop in at my very short (10-15 minutes!) lightning talk about libguestfs, virt-builder, virt-sparsify and more.

Room: H.2215 (Ferrer)
Day: Sunday 2nd Feb 2014
Start: 16:00

Leave a comment

Filed under Uncategorized

xz plugin for nbdkit

I’ve now written an xz plugin for nbdkit (previous discussion on this blog).

This is useful if you’re building up a library of xz-compressed disk images using virt-sparsify and xz, and you want to access them without having to uncompress them.

I certainly learned a lot about the xz file format and liblzma this weekend …

The xz file format consists of multiple streams (but usually one). Each stream contains zero or more blocks of compressed data, followed by an “index”. Like zip, everything in an xz file happens from the end, so the block index is at the end of the stream (this allows xz files to be streamed when writing without needing any reverse seeks).

Crucially the index contains the offset of each block both in the actual xz file and in the uncompressed data, so once you’ve read the index from a file you can find the position of any uncompressed byte and seek to the beginning of that block and read the data. Random access!

Preparing xz files correctly is important in order to be able to get good random access performance with low memory overhead:

$ xz --list /tmp/winxp.img.xz
Strms  Blocks   Compressed Uncompressed  Ratio  Check   Filename
    1     384  2,120.1 MiB  6,144.0 MiB  0.345  CRC64   /tmp/winxp.img.xz

A file with lots of small blocks like the above (16 MB block size) is relatively easy to seek inside. At most 16 MB of data has to be uncompressed to reach any byte.

Perhaps ironically, if your machine has lots of free memory then xz appears to choose a large block size, resulting in some one-block files. Here’s the same file when I originally compressed it for my guest library:

$ xz --list guest-library/winxp.img.xz
Strms  Blocks   Compressed Uncompressed  Ratio  Check   Filename
    1       1  2,100.0 MiB  6,144.0 MiB  0.342  CRC64   guest-library/winxp.img.xz

So unfortunately you may need to recompress some of your xz files using the new xz --block-size option:

$ xz --best --block-size=$((16*1024*1024)) winxp.img

Here’s how you use the new nbdkit xz plugin:

$ nbdkit plugins/nbdkit-xz-plugin.so file=winxp.img.xz
$ guestfish --ro -a nbd://localhost -i

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

Operating system: Microsoft Windows XP
/dev/sda1 mounted on /

><fs> ll /
total 1573209
drwxrwxrwx  1 root root       4096 Apr 16  2012 .
drwxr-xr-x 23 1000 1000       4096 Jun 24 13:57 ..
-rwxrwxrwx  1 root root          0 Oct 11  2011 AUTOEXEC.BAT
-rwxrwxrwx  1 root root          0 Oct 11  2011 CONFIG.SYS
drwxrwxrwx  1 root root       4096 Oct 11  2011 Documents and Settings
-rwxrwxrwx  1 root root          0 Oct 11  2011 IO.SYS
-rwxrwxrwx  1 root root          0 Oct 11  2011 MSDOS.SYS
[...]

5 Comments

Filed under Uncategorized

Build a guest library

I have this exported over NFS to all my local machines:

$ ls -lh /mnt/media/guest-library/
total 12G
-rw-r--r--. 1 rjones rjones 971M Apr 16 23:38 debian5x64.img.xz
-rw-r--r--. 1 rjones rjones 250M Jul 26 23:09 f16x32.img.xz
-rw-r--r--. 1 rjones rjones 809M Jul 26 20:29 f17x64.img.xz
-rw-r--r--. 1 rjones rjones  59M Jul  6 22:19 freedos11.img.xz
-rw-r--r--. 1 rjones rjones 379M Jul  7 20:11 opensuse1113x64.img.xz
-rw-r--r--. 1 rjones rjones 349M Jul  6 22:31 plan9.img.xz
-rw-rw-r--. 1 rjones rjones  126 Apr 16 17:22 README
-rw-r--r--. 1 rjones rjones 384M Apr 18 00:19 rhel3x64.img.xz
-rw-r--r--. 1 rjones rjones 471M Apr 18 00:15 rhel4x64.img.xz
-rw-r--r--. 1 rjones rjones 848M Apr 16 17:02 rhel5epelx64.img.xz
-rw-r--r--. 1 rjones rjones 768M Jul  7 07:21 sl60x64.img.xz
-rw-r--r--. 1 rjones rjones 148M May 23 16:08 ubuntu1204ppc.img.xz
-rw-r--r--. 1 rjones rjones 1.1G Apr 16 17:09 win2003.img.xz
-rw-r--r--. 1 rjones rjones 3.4G Jun  1 10:46 win8-preview.img.xz
-rw-r--r--. 1 rjones rjones 2.1G Apr 16 20:28 winxp.img.xz

Having a library of images which are (relatively) small like this is great for testing. If someone gives me a bug and it happens to affect one of the guests I have saved, then I can just unpack that guest, virt-resize it to the right size and off I go, and that’s usually a lot quicker than installing a new guest, particularly when you’ve got only slow ADSL.

Adding a new guest is easy using virt-sparsify and xz. Note I’ve set the temporary directory (TMPDIR) to somewhere with plenty of space.

$ sudo TMPDIR=$(pwd) virt-sparsify /dev/vg_pin/F18Rawhidex32 f18rawhidex32.img
Create overlay file to protect source disk ...
Examine source disk ...
 100% ⟦▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓⟧ 00:00
Fill free space in /dev/vda1 with zero ...
 100% ⟦▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓⟧ --:--
Fill free space in /dev/vg_f18rawhidex32/lv_root with zero ...
 100% ⟦▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓⟧ --:--
Clearing Linux swap on /dev/vg_f18rawhidex32/lv_swap ...
 100% ⟦▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓⟧ 00:00
Copy to destination and make sparse ...

Sparsify operation completed with no errors.  Before deleting the old 
disk, carefully check that the target disk boots and works correctly.

$ ll -h f18rawhidex32.img
-rw-r--r--. 1 root root 10G Jul 27 14:42 f18rawhidex32.img
$ du -sh f18rawhidex32.img
3.0G	f18rawhidex32.img
$ xz -T 0 --best f18rawhidex32.img
$ ll -h f18rawhidex32.img.xz
-rw-r--r--. 1 rjones rjones 667M Jul 27 14:42 f18rawhidex32.img.xz

2 Comments

Filed under Uncategorized

Using virt-sparsify and xz, we can really compress VMs for storage

17G	debian5x64.img.orig

The original image has about 4.7 GB of data, plus a large swap partition, according to virt-df:

$ virt-df -a debian5x64.img.orig -h
Filesystem                        Size       Used  Available  Use%
debian5x64.img.orig:/dev/sda1     322M        66M       239M   21%
debian5x64.img.orig:/dev/debian5x64.home.annexia.org/home
                                  3.4G       359M       2.9G   11%
debian5x64.img.orig:/dev/debian5x64.home.annexia.org/root
                                  320M       301M       2.4M   95%
debian5x64.img.orig:/dev/debian5x64.home.annexia.org/tmp
                                  300M       8.2M       276M    3%
debian5x64.img.orig:/dev/debian5x64.home.annexia.org/usr
                                  3.4G       2.0G       1.2G   60%
debian5x64.img.orig:/dev/debian5x64.home.annexia.org/var
                                  2.6G       2.0G       536M   76%

Using virt-sparsify, all unused space in the image is made sparse. The most recent version can sparsify swap partitions too:

4.6G	debian5x64.img

xz --best -T 0 reduces the final image to under a gigabyte:

971M	debian5x64.img.xz

5 Comments

Filed under Uncategorized

New in virt-sparsify

New in libguestfs 1.17.18, virt-sparsify can now find and sparsify even more space in your guests — Linux swap partitions are detected and zeroed (though not if they contain hibernation data).

The --zero option can be used to remove unwanted partitions.

The new version is also faster and uses less temporary space during the first stage.

Leave a comment

Filed under Uncategorized

New tool: virt-sysprep

New in libguestfs 1.13.19 is a tool called virt-sysprep which makes it easier to clone virtual machines.

The new tool isn’t a complete clone tool, but you can easily combine it with other tools like dd, the old virt-clone, virt-resize and virt-sparsify to clone virtual machines.

6 Comments

Filed under Uncategorized