a-fedora-appliance updated for supermin 5

a-fedora-appliance is a supermin demonstration Fedora appliance.

I have scratch-built a Fedora RPM here which is just a 235K download but contains (by magic!) a fully bootable Fedora appliance. After installing the RPM in Fedora 20, do the following to boot the virtual machine:

# boot-a-fedora-appliance

Note that the scratch build will only last in Koji for a few days. After that you’ll have to follow the README file included in the source.

Tip: Old supermin, new libguestfs and v.v.

Problem:

You want to compile libguestfs ≥ 1.25.38, but your distro only has old supermin 4.

Solution:

Compile supermin from source. Note do not install it!

git clone https://github.com/libguestfs/supermin supermin5
cd supermin5
./autogen.sh
make

Create a file called localenv in the libguestfs build directory with the following content:

export SUPERMIN=/path/to/supermin5/src/supermin

and a file localconfigure containing:

source localenv
./configure "$@"

chmod +x localconfigure

Rebuild libguestfs as normal, except you use ./localconfigure instead of ./configure

---

Problem:

You want to compile libguestfs ≤ 1.24, but you’ve installed new supermin 5.

Solution:

Compile supermin 4 from source. Note do not install it!

git clone -b supermin-4.x \
    https://github.com/libguestfs/supermin supermin4
cd supermin4
./autogen.sh
make

Create a file called localenv in the libguestfs build directory with the following content:

export SUPERMIN=/path/to/supermin4/src/supermin
export SUPERMIN_HELPER=/path/to/supermin4/helper/supermin-helper

and a file localconfigure containing:

source localenv
./configure "$@"

chmod +x localconfigure

Rebuild libguestfs as normal, except you use ./localconfigure instead of ./configure

Supermin version 5

Recently myself with help from Pino Toscano and Hilko Bengen rewrote supermin [git repo] to make it more featureful and robust. Supermin is a clever tool that lets you distribute very tiny (< 100K) Linux appliances, which are reconstituted to full appliances just before they run.

There’s an Adam West-era Batman film where the UN Security Council is dehydrated by The Penguin, leaving little piles of powder. Batman (sorry to reveal the ending here!) saves the day by adding water and rehydrating the UN. Supermin works in much the same way — by observing that you don’t need to store (eg) /bin/bash in the appliance, since /bin/bash already exists on the host. By just storing a pointer to /bin/bash instead, you get the amazing compression ratios. Just add water.

Packages

Supermin 4 (the previous version) stored a list of filenames that would be copied into the full appliance. This was somewhat fragile if the host distro changed, eg. moving files around.

Supermin 5 stores the list of package names, and it resolves the dependencies and filenames using the RPM/dpkg/other package manager database just before reconstituting the full appliance. This solves the fragility problem completely. It also means we are now able to split libguestfs dependencies as described here.

Locking and caching

The other part of supermin which made it difficult to use in practice was locking, or rather the absence of locking. If you wanted to use supermin 4 from multiple threads, or multiple processes, you had the problem that they could race to build the full appliance potentially overwriting each other’s output. So you had to implement some sort of locking in the higher layers. And you also had to work out yourself if the full appliance needed to be rebuilt at all or if you could use a cached copy.

In supermin 5, locking and caching is now managed entirely by supermin itself and all the caller has to do is to pass two simple command line arguments:

supermin --build \
    --if-newer \
    --lock /run/lock/supermin.lock \
    [...]

and supply the location of a lock file. This also works if the application is multithreaded, and if the application wants to build multiple appliances (you supply a different lock file per appliance).

Chroots

Supermin used to be called “febootstrap”. That was a bad, confusing name. However one feature that febootstrap had was the ability to build chroots, which we dropped in supermin. It turns out that lots of people liked building chroots for containers — indeed it is even the recommended way to build RHEL/CentOS 6 chroots for Docker.

Supermin 5 can build chroots again by selecting the --format chroot output format. Here’s how you could build a chroot from scratch:

$ supermin --prepare -o /tmp/supermin.d \
    bash coreutils
$ supermin --build --format chroot \
    -o /tmp/appliance.d /tmp/supermin.d

Tarballs, hostfiles, excludefiles

Supermin has always allowed you to customize the full appliance by specifying extra static files or host-copied files in the supermin appliance. However in supermin 4 you had to use a specially formatted cpio file to do this.

In supermin 5 you just use a regular tarball, eg:

$ tar zcf /tmp/supermin.d/init.tar.gz ./init

Supermin 5 also lets you specify files to be copied from the host. Just create a list of wildcards, one per line:

$ cat > /tmp/supermin.d/hostfiles <<EOF
/usr/share/augeas/lenses/*.aug
EOF

Additionally you can specify a list of files to be excluded from the full appliance, which is useful for dropping documentation and other irrelevant stuff:

$ cat > /tmp/supermin.d/excludefiles <<EOF
-/usr/share/doc/*
-/usr/share/info/*
-/usr/share/man/*
EOF

More information

For more information about supermin, read the supermin(1) manual page online. You can also build it from the git repository, and it is available in Fedora Rawhide and Debian/experimental.

Supermin 5 will be required for libguestfs ≥ 1.26 when it is released shortly.

Half-baked ideas: Kernel modules-in-a-file

For more half-baked ideas, see the ideas tag.

The Linux kernel:

/boot/vmlinuz-3.11.9-200.fc19.x86_64

5 MB of highly optimized code that runs the world’s computers. You can copy it around and boot it on different hardware.

No wait, that’s all wrong. I’m forgetting about the modules. There are another 2832 files in /lib/modules/3.11.9-200.fc19.x86_64 and without them this kernel will hardly function.

It’s nice to be able to copy the kernel around — to compile it on fast x86-64 and run it on slow ARM, or to run the host kernel in a VM, but all those module files make everyone’s lives a lot more complex.

So my half-baked idea is this:

Combine all the modules into a single file, let’s call it /boot/modules-3.11.9-200.fc19.x86_64. This file might start off as an ext4 filesystem which can be loopback-mounted on /lib/modules. Eventually I’d hope that the module tools could load modules from the file directly.

Now you can happily cross-compile and copy just two kernel files from x86 to ARM or into your VMs, and that’s a lot less hassle.

febootstrap is now called supermin

The new supermin home page is here, tarballs are available here, and the new git repository is here.

An explanation of supermin appliances

I wrote an explanation of supermin appliances.

libguestfs launch times

Indulge me while a make a “note to self” about efforts to reduce the time taken by guestfs_launch which boots up the libguestfs appliance.

Time (s)	Operation
2s	Create supermin appliance: This has crept up over time from originally taking about 1/5th of a second to around 2s. Needs attention. Fixed see this note about cpio blocksize and update below.
2-3s	qemu startup: The time is mainly spent reading in the large -kernel and particularly -initrd files specified on the command line. The released qemu code is quite rubbish, but luckily kraxel beat me to fixing the problem with this patch.
3s	BIOS waits for keypress: As discussed yesterday, I’ve posted a patch. In the meantime the qemu devs have abandoned the old bochs BIOS for SeaBIOS, which I haven’t tested yet,
3s	Kernel boot time: Not very many easy wins here, since the kernel is already pretty efficient. We could try to remove some busy waits and sleeps — for example the kernel waits ¼s for the serial ports, which we don’t use.
1-2s	Userspace boot time: This is mainly time spent on udev and partition detection. I have never really understood why the kernel needs to pause so long on partition detection. Not much easy meat here, but improving the speed of udev in itself would be worthwhile.

I spotted a nice tool for turning strace output into timelines.

Update — bash globbing

Having solved the cpio problem, the largest bottleneck in creating the supermin appliance becomes globbing.

It’s probably not a well-known fact, but if you do:

ls *.c *.h

then bash reads the directory twice. It treats the two globs on the command line as completely separate entities. This is not so bad for a few globs, but when we make the supermin appliance we need to do over 120 globs, which takes bash about 0.8s to complete, contributing about 10% to the overall launch time. Bash is literally reading the same directory over and over again, 50 or more times.

At the moment it’s not obvious to me how to solve this.

Supermin appliance – now in febootstrap

Q: Can you fit a bootable Fedora distribution into 100 kilobytes?
A: You bet!*
* by cheating … read on.

Take an ordinary Fedora appliance as made by febootstrap or appliance-creator. The appliance image is big because it contains copies of programs (/bin/bash) and libraries (/lib/libc.so), the kernel and kernel modules. It needs to, to make it self-contained.

But a technique we’ve been using for a few months in libguestfs is to say: we’re booting this appliance on a Fedora host. Let’s strip out all those programs and libraries from the appliance, and we’ll add them back from the host just before we launch it.

I called such appliances “supermin appliances”, and now I’ve ported the functionality from libguestfs into febootstrap so everyone can use it.

A supermin appliance is really small:

-rw-rw-r-- 1 rjones rjones 14K 2009-10-22 12:43 hostfiles.txt
-rw-rw-r-- 1 rjones rjones 87K 2009-10-22 12:43 supermin.img

yet it’s fully bootable, given the right sort of host:

It’s really easy to use the new febootstrap to make your own super-small supermin appliances.

We’ll start with this small shell script to make a supermin appliance:

#!/bin/sh -
distro=rawhide
febootstrap -i bash -i coreutils $distro fedora
febootstrap-minimize fedora
cat > init <<EOF
#!/bin/sh
echo Starting /init script ...
PATH=/sbin:/usr/sbin:$PATH
touch /etc/fstab
mount -t proc /proc /proc
mount -t sysfs /sys /sys
exec bash -i
EOF
febootstrap-install fedora init /init 0755 root.root
rm init
# Create the supermin appliance.
febootstrap-to-supermin fedora supermin.img hostfiles.txt
# Create the ordinary appliance just for comparison.
febootstrap-to-initramfs fedora > ordinary.img

You will need to use febootstrap >= 2.5, and set the distro variable so it exactly matches your base Fedora system (eg. set distro=fedora-11).

The script below can be used to boot the appliance, and it’s what I used to get the screenshot above.

#!/bin/sh -
time febootstrap-supermin-helper \
    supermin.img hostfiles.txt kernel initrd
qemu-system-x86_64 -m 1024 -kernel kernel -initrd initrd