Tag Archives: ubuntu

Analysis of the size of libguestfs dependencies

In libguestfs ≥ 1.26 we are going to start splitting the package up into smaller dependencies. Since the full libguestfs package has lots of dependencies because it has to be able to process lots of obscure filesystems, the question is how best to split up the dependencies? We could split off, say, XFS support into a subpackage, but how do we know if that will save any space?

Given the set of dependencies, we want to know the incremental cost of adding another dependency.

We can get an exact measure of this by using supermin to build a chroot containing the set of dependencies, and a second chroot containing the set of dependencies + the additional package. Then we simply compare the sizes of the two chroots. The advantage of using supermin is that the exact same script [see end of posting] will work for Fedora and Debian/Ubuntu since supermin hides the complexity of dealing with the different package managers through its package manager abstraction.

The results of this, using the libguestfs appliance dependencies, on Fedora 20, sorted by dependency size, with my comments added:

  1. gdisk adds 25420 KB

    This is a surprising result in first place, since gdisk is a fairly small, unassuming C++ program (only ~11KLoC). My initial thought was it must be something to do with being written in C++, but I tested that and it’s not true. The real problem is that gdisk depends on libicu (a Unicode library) which adds 24.6 MB to the appliance. [Note: this issue has been fixed in Rawhide.]

  2. lvm2 adds 19432 KB

    The default disk layout of many Linux distros uses LVM so this and similar dependencies have to stay in base libguestfs.

  3. binutils adds 16604 KB

    This is a sorry tale. The one file we use from binutils is /usr/bin/strings (33KB). Unfortunately this single binary pulls in a huge dependency (even worse, it’s a development package, and this causes problems on production systems). I don’t really understand why strings is included in binutils.

  4. gfs2-utils adds 9648 KB
  5. zfs-fuse adds 5208 KB

    Split off in the proposed reorganization.

  6. ntfsprogs adds 4572 KB
  7. e2fsprogs adds 4312 KB

    Most Linux distros use ext4, and we want to support Windows out of the box, so these are included in base libguestfs.

  8. xfsprogs adds 3532 KB

    Split off in the proposed reorganization.

  9. iproute adds 3180 KB

    We use /sbin/ip to set up the network card inside the appliance. It’s a shame this “better” replacement for ifconfig is so large.

  10. tar adds 2896 KB
  11. btrfs-progs adds 2800 KB
  12. openssh-clients adds 2428 KB
  13. parted adds 2420 KB
  14. jfsutils adds 1668 KB
  15. genisoimage adds 1644 KB
  16. syslinux-extlinux adds 1420 KB
  17. augeas-libs adds 1404 KB
  18. iputils adds 1128 KB
  19. reiserfs-utils adds 1076 KB
  20. mdadm adds 1032 KB
  21. strace adds 976 KB
  22. lsof adds 972 KB
  23. vim-minimal adds 912 KB
  24. rsync adds 812 KB
  25. libldm adds 616 KB
  26. psmisc adds 592 KB
  27. nilfs-utils adds 520 KB
  28. hfsplus-tools adds 480 KB

The test script used to produce these results:

#!/bin/bash -

# NB: For this program to work, you must have the following
# packages (or as many as possible) installed locally.
pkgs='acl attr augeas-libs bash binutils bsdmainutils btrfs-progs
bzip2 coreutils cpio cryptsetup cryptsetup-luks diffutils dosfstools
e2fsprogs extlinux file findutils gawk gdisk genisoimage gfs2-utils
grep grub grub-pc gzip hfsplus hfsplus-tools hivex iproute iputils
jfsutils kernel kmod less libaugeas0 libcap libcap2 libhivex0 libldm
libpcre3 libselinux libsystemd-id128-0 libsystemd-journal0 libxml2
libyajl2 linux-image lsof lsscsi lvm2 lzop mdadm module-init-tools
mtools nilfs-utils ntfs-3g ntfsprogs openssh-clients parted pcre
procps procps-ng psmisc reiserfs-utils reiserfsprogs rsync scrub sed
strace syslinux syslinux-extlinux systemd sysvinit tar udev ufsutils
util-linux util-linux-ng vim-minimal vim-tiny xfsprogs xz xz-utils
yajl zerofree zfs-fuse'

# These are the packages (from the above list) that we want to test.

# Helper function to construct an appliance and see how big it is.
function appliance_size
    set -e
    supermin --prepare -o /tmp/supermin.d "$@" >&/dev/null
    supermin --build -f chroot -o /tmp/appliance.d \
      /tmp/supermin.d >&/dev/null
    du -s /tmp/appliance.d | awk '{print $1}'

# Construct entire appliance to see how big that would be.
totalsize=`appliance_size $pkgs`

# Remove each package from the list in turn, and find out
# how much extra that package contributes.
for p in $testpkgs; do
    for o in $pkgs; do
        if [ $o != $p ]; then opkgs="$opkgs $o"; fi
    size=`appliance_size $opkgs`
    extra=$(($totalsize - $size))

    echo $p adds $extra KB

1 Comment

Filed under Uncategorized

Virt-builder making VMs in around 3̶0̶ 16 seconds

The newest upstream virt-builder can now build simple VMs in around 30 seconds (this is on a machine with a hard drive, it’s faster if you have SSDs):

$ virt-builder ubuntu-12.04
[   0.0] Downloading: file:///home/rjones/d/libguestfs/builder/website/ubuntu-12.04.xz
[   1.0] Creating disk image: ubuntu-12.04.img
[   1.0] Uncompressing: file:///home/rjones/d/libguestfs/builder/website/ubuntu-12.04.xz
[   9.0] Opening the new disk
[  33.0] Setting a random seed
[  33.0] Random root password: z6T5zQiqIIy0yZfA [did you mean to use --root-password?]
[  33.0] Finishing off
Output: ubuntu-12.04.img
Total usable space: 2.9G
Free space: 2.1G (73%)

Even installing packages in the VM, it’s still taking under 1 minute:

$ virt-builder ubuntu-12.04 --install nmap
[   0.0] Downloading: file:///home/rjones/d/libguestfs/builder/website/ubuntu-12.04.xz
[   1.0] Creating disk image: ubuntu-12.04.img
[   1.0] Uncompressing: file:///home/rjones/d/libguestfs/builder/website/ubuntu-12.04.xz
[  10.0] Opening the new disk
[  34.0] Setting a random seed
[  34.0] Random root password: Mt3kxbEsBqumG6bR [did you mean to use --root-password?]
[  34.0] Installing packages: nmap
[  53.0] Finishing off
Output: ubuntu-12.04.img
Total usable space: 2.9G
Free space: 2.0G (70%)

The major changes are: It will use pxzcat (parallel xzcat) if available, and it will bypass the virt-resize step if you don’t specify the --size option (or if you make sure the templates are the right size to start with).


On a machine with an Intel SSD and parallel xzcat installed:

$ virt-builder ubuntu-12.04
[   0.0] Downloading: file:///home/rjones/d/libguestfs/builder/website/ubuntu-12.04.xz
[   1.0] Creating disk image: ubuntu-12.04.img
[   1.0] Uncompressing: file:///home/rjones/d/libguestfs/builder/website/ubuntu-12.04.xz
[  12.0] Opening the new disk
[  16.0] Setting a random seed
[  16.0] Random root password: JdIsqD5QB64yImYL [did you mean to use --root-password?]
[  16.0] Finishing off
Output: ubuntu-12.04.img
Total usable space: 2.9G
Free space: 2.1G (73%)

Leave a comment

Filed under Uncategorized

Tip: Create a libvirt guest using virt-builder

virt-builder and virt-install together make it easy to create libvirt guests in just a few minutes.

Before you start, read the installation notes for the OS you’re about to install. They may contain essential tips for a successful installation:

$ virt-builder --notes ubuntu-13.10

Note I’m running this as root, but that’s only necessary because I want to write the guest to a logical volume. Normally you don’t (and shouldn’t) run virt-builder as root.

# lvcreate -L 50G -n builder-ubuntu1310 /dev/vg_data
# virt-builder ubuntu-13.10 \
    --output /dev/vg_data/builder-ubuntu1310 \
    --hostname builder-ubuntu1310 \
    --install build-essential,debhelper
[   0.0] Downloading: http://libguestfs.org/download/builder/ubuntu-13.10.xz
[   1.0] Uncompressing: http://libguestfs.org/download/builder/ubuntu-13.10.xz
[  23.0] Running virt-resize to expand the disk
[  99.0] Opening the new disk
[ 103.0] Setting a random seed
[ 103.0] Setting the hostname: builder-ubuntu1310
[ 103.0] Random root password: mOQ8YdwKuxkMlpOU [did you mean to use --root-password?]
[ 103.0] Installing packages: build-essential debhelper
[ 423.0] Finishing off
Output: /dev/vg_data/builder-ubuntu1310
Total usable space: 48.2G
Free space: 47.1G (97%)
# virt-install --import \
    --name builder-ubuntu1310 --ram 4096 \
    --disk path=/dev/vg_data/builder-ubuntu1310,format=raw

(Yes, I forgot to use the --root-password option …)


Filed under Uncategorized

libguestfs packages for Ubuntu

This directory contains experimental up to date libguestfs packages for Ubuntu 12.10. You should be able to install them by adding this line to /etc/apt/sources.list:

deb http://libguestfs.org/download/binaries/ubuntu1210-packages/ /

You will need to also:

sudo chmod 0644 /boot/vmlinuz-*

because of this Ubuntu bug.

Let me know if the packages work. Also what other versions of Ubuntu I should be building them for.


Filed under Uncategorized

Installing Ubuntu 12.04 PowerPPC on qemu

Fedora gave up building on ppc as a primary architecture a while back [edit: see comments], but Ubuntu has a working ppc build. This is useful for testing software because it’s a big endian architecture, and therefore breaks some assumptions made by software that has only seen an Intel (little endian) architecture.

Fortunately it’s very simple to install Ubuntu/ppc as a qemu guest. Here is how I did it:

  1. Download mini.iso from here.
  2. Compile qemu from git (it’s easy!) so you have a qemu-system-ppc binary with a working bios.
  3. Create a virtual hard disk: truncate -s 10G disk.img
  4. Boot the ISO: ./qemu-system-ppc -m 1024 -hda disk.img -cdrom mini.iso -boot d
  5. At the first prompt, type install and go through the installation.

At the end of the installation, it won’t install a boot loader, so the guest won’t be bootable without an external kernel and initrd. This is easy to arrange:

$ guestfish --ro -a disk.img -m /dev/sda2 \
    download /vmlinux vmlinux : \
    download /initrd.img initrd.img

With the external files vmlinux and initrd.img you can now boot your guest:

$ ./qemu-system-ppc -m 1024 \
    -hda disk.img \
    -kernel vmlinux -initrd initrd.img \
    -append "ro root=/dev/sda3"


Filed under Uncategorized

Debian/Ubuntu package for Oz (an excellent VM builder)

Oz is an excellent virtual machine builder. I reviewed it here.

This is a temporary Debian (and possibly Ubuntu) package for Oz:


I’ll put up a more permanent site for Debian packages later when we’ve decided where to host them.

Note there is no separate source archive of this right now. The source is upstream git + the Debian patch that I posted on the mailing list (but the mailing list archives seem to be broken — fixing it). You can build the exact same package yourself by checking out git, applying the Debian patch, and doing:

debuild -i -uc -us -b


Filed under Uncategorized

libguestfs packages for Ubuntu 10.04LTS

Here you can find libguestfs packages for Ubuntu 10.04LTS:


Make sure you read the README first.

For a list of places to find binaries for other versions and distros, see here.

Leave a comment

Filed under Uncategorized

Ubuntu packages for libguestfs 1.12.0

The Ubuntu packages are here. Read the README file before trying to install them.

Leave a comment

Filed under Uncategorized

New libguestfs packages for Debian and Ubuntu

libguestfs 1.11.8 binary packages for Debian and for Ubuntu.

The separate insmod.static binary is no longer needed (thanks Hilko Bengen).


Filed under Uncategorized

libguestfs build — an open ended problem

librarian made a very true observation (Google translate) about libguestfs. It’s a Swiss army chainsaw, but it’s damn hard to build from source.

With RHEL and Fedora I’ve made it my aim that no one should need to build libguestfs from source, because we offer the highest quality packages with every feature compiled in. I also build Debian and Ubuntu packages when I can and until someone steps up to do that.

But why is libguestfs a difficult package to build?

The primary reason is that we package up, make an API for, and rigorously test, something like 200 different Linux packages. Essentially if you use (say) the guestfs_part_* API then in the background you’re using parted. If you’re using another API, you might be using e2fsck or resize2fs or lvm or grep or file or the kernel or any one of dozens of other programs. And to compound the problem, we don’t just “ship and forget”. We test these programs, and if they break, then we break. Our test suite has about 600 different tests and takes 2 hours to run.

And we test against Fedora Rawhide. The latest and buggiest.

Consequently we hit all the new bugs. Just today I hit a Linux 3.0 bug and another kernel/ftrace bug. Two weeks ago it was a bug in the file command, another bug in udev on Debian, and you can never exclude the possibility of stupidity by Ubuntu kernel maintainers.

It’s routine that I discover qemu, kernel and other bugs for the first time, because often a libguestfs build in Koji is the first build that boots up and runs the new software.

So what’s my point? It would be good if the Fedora kernel and qemu maintainers didn’t just push out a new package, but they tested that one can run inside the other. But while that would improve the situation for me, the real problem is that integrating software is hard, and it’s unfortunate that libguestfs has got into a situation where we are the first people to integrate and run Rawhide.

Leave a comment

Filed under Uncategorized