New libguestfs packages for Debian and Ubuntu

libguestfs 1.11.8 binary packages for Debian and for Ubuntu.

The separate insmod.static binary is no longer needed (thanks Hilko Bengen).

libguestfs build — an open ended problem

librarian made a very true observation (Google translate) about libguestfs. It’s a Swiss army chainsaw, but it’s damn hard to build from source.

With RHEL and Fedora I’ve made it my aim that no one should need to build libguestfs from source, because we offer the highest quality packages with every feature compiled in. I also build Debian and Ubuntu packages when I can and until someone steps up to do that.

But why is libguestfs a difficult package to build?

The primary reason is that we package up, make an API for, and rigorously test, something like 200 different Linux packages. Essentially if you use (say) the guestfs_part_* API then in the background you’re using parted. If you’re using another API, you might be using e2fsck or resize2fs or lvm or grep or file or the kernel or any one of dozens of other programs. And to compound the problem, we don’t just “ship and forget”. We test these programs, and if they break, then we break. Our test suite has about 600 different tests and takes 2 hours to run.

And we test against Fedora Rawhide. The latest and buggiest.

Consequently we hit all the new bugs. Just today I hit a Linux 3.0 bug and another kernel/ftrace bug. Two weeks ago it was a bug in the file command, another bug in udev on Debian, and you can never exclude the possibility of stupidity by Ubuntu kernel maintainers.

It’s routine that I discover qemu, kernel and other bugs for the first time, because often a libguestfs build in Koji is the first build that boots up and runs the new software.

So what’s my point? It would be good if the Fedora kernel and qemu maintainers didn’t just push out a new package, but they tested that one can run inside the other. But while that would improve the situation for me, the real problem is that integrating software is hard, and it’s unfortunate that libguestfs has got into a situation where we are the first people to integrate and run Rawhide.

Building libguestfs from source tarballs on Debian and Ubuntu

It is now possible to build libguestfs from the source tarballs on Debian and Ubuntu (into .deb packages which can be installed and managed by apt/aptitude). You will need libguestfs ≥ 1.11.6.

It’s not exactly obvious how, so this guide describes how to do it.

You’ll first need to install febootstrap 3.3, which you can either build from source in a similar way, or simpler you can install the binary package from here.

You will also need pretty recent Debian/Ubuntu. Suggested source repositories are here. In particular kernel ≥ 2.6.36 is absolutely required on the host. Earlier kernel versions than this won’t work.

Debian and Ubuntu do not supply /sbin/insmod.static. For the suggested workaround, read this.

You will also need to install several packages, but the first time you run the debuild command, it should tell you exactly what you are missing.

Grab the libguestfs source tarball ≥ 1.11.6 from the website, and unpack it:

$ zcat libguestfs-1.11.6.tar.gz | tar xf -
$ cd libguestfs-1.11.6

You can now run debuild with the right collection of options to build. The first time you do this, it will probably tell you that you need to install a bunch of build-time dependencies.

$ debuild -i -us -uc -b 2>&1 | tee /tmp/log

If it doesn’t work you’ll need to go back over the log file.

Once that works, in the parent directory (ie. above the build directory) you should find several .deb files:

$ cd ..
$ ls -l
$ ls -l
total 5280
-rw-r--r--  1 rjones rjones  347640 May 12 14:27 guestfish_1.11.6-1_amd64.deb
-rw-r--r--  1 rjones rjones  181368 May 12 14:27 guestmount_1.11.6-1_amd64.deb
-rw-r--r--  1 rjones rjones  941416 May 12 14:27 libguestfs0_1.11.6-1_amd64.deb
-rw-r--r--  1 rjones rjones 1235918 May 12 14:27 libguestfs0-dbg_1.11.6-1_amd64.deb
drwxr-xr-x 38 rjones rjones    4096 May 12 14:27 libguestfs-1.11.6
-rw-r--r--  1 rjones rjones  561346 May 12 14:28 libguestfs_1.11.6-1_amd64.build
-rw-r--r--  1 rjones rjones    3637 May 12 14:27 libguestfs_1.11.6-1_amd64.changes
-rw-r--r--  1 rjones rjones  408490 May 12 14:27 libguestfs-dev_1.11.6-1_amd64.deb
-rw-r--r--  1 rjones rjones  364450 May 12 14:27 libguestfs-doc_1.11.6-1_all.deb
-rw-r--r--  1 rjones rjones  454984 May 12 14:27 libguestfs-perl_1.11.6-1_amd64.deb
-rw-r--r--  1 rjones rjones  578720 May 12 14:27 libguestfs-tools_1.11.6-1_amd64.deb
-rw-r--r--  1 rjones rjones  262584 May 12 14:27 python-guestfs_1.11.6-1_amd64.deb

You can just install these using dpkg -i or however you would normally install .deb packages.

Finally make sure everything is working by running libguestfs-test-tool.

libguestfs 1.9.6 packages for Debian and Ubuntu

Debian packages and Ubuntu packages. Note that 1.9.6 is a development version.

New (Year’s) libguestfs tools: virt-copy-in, virt-copy-out, virt-tar-in, virt-tar-out

One aim with libguestfs development is to make easy and common file operations easy. Although you can already upload and download files into virtual machines using guestfish commands, is there a way to make this common operation easier to discover?

One way is to add more virt commands, which I’ve found that users have least difficulty discovering because they are on the website, autocompleted when you hit virt-[tab], and listed as separated manual pages.

So today I added four more commands for uploading and downloading: virt-copy-in, virt-copy-out, virt-tar-in, virt-tar-out.

The way you use them is very simple:

$ mkdir homes
$ virt-copy-out -d Fedora14 /home homes/

$ virt-tar-out -d Fedora14 /home - | \
    gzip --best > homes.tar.gz

These commands are just small shell script wrappers around guestfish, but I hope they make common things a little bit easier.

You can get these new commands from Fedora Rawhide, or as binaries for Debian or Ubuntu.

libguestfs 1.8.0 released

libguestfs is a set of tools and a library for accessing, creating and modifying the contents of virtual machines and disk images.

Source — Fedora 14 — Ubuntu packages — Debian packages — ArchLinux packages (thanks Thomas S. Hatch)

The full release notes for 1.8 are here.

Tip: Creating throwaway appliances with febootstrap

This is an update to a previous posting, but using the new febootstrap 3.x cross-distro toolchain to make building supermin appliances even simpler.

Firstly we create a supermin appliance containing a few packages and their dependencies. Note that I’m not minimizing this appliance so it’s a bit bigger (3.1MB!) than the ones we would generate in reality:

$ mkdir supermin.d
$ febootstrap --names 'bash' 'coreutils' -o supermin.d
febootstrap: warning: some host files are unreadable by non-root
febootstrap: warning: get your distro to fix these files:
	/sbin/unix_update
	/usr/libexec/pt_chown
	/usr/sbin/build-locale-archive
	/usr/sbin/glibc_post_upgrade.x86_64
	/usr/sbin/tzdata-update
$ ls -lh supermin.d/
total 3.1M
-rw-rw-r--. 1 rjones rjones 2.7M Dec 10 18:23 base.img
-rw-rw-r--. 1 rjones rjones 462K Dec 10 18:23 hostfiles

The purpose of these two files is explained in the febootstrap documentation.

This won’t boot without a /init script, and we can easily provide one:

$ cat init
#!/bin/bash -
echo Welcome to my world
bash -i
$ chmod +x init
$ echo init | cpio -o -H newc --quiet > supermin.d/init.img

The 3 files in supermin.d/ are a supermin appliance, and could be packaged up in a Fedora, Debian or Ubuntu package.

When you actually want to come and launch this appliance, you use febootstrap-supermin-helper to reconstruct the appliance:

$ febootstrap-supermin-helper -f ext2 supermin.d x86_64 \
    kernel initrd root

(Note that “kernel”, “initrd” and “root” are output files in that command)

You’d usually arrange for those files to be cached, since febootstrap-supermin-helper takes a few seconds to run (8 seconds on my laptop) and by caching it you can get reconstruction time down to a fraction of a second.

Now to boot, run qemu or qemu-kvm like this:

$ qemu-kvm -kernel kernel -initrd initrd -hda root

After a few seconds you’ll get to the shell:

Ubuntu packages for libguestfs

You can download Ubuntu packages (*.deb files) for libguestfs. Don’t forget to read the README file.

Previously: Debian packages.

libguestfs Debian packages

Up to date Debian packages (*.deb) for libguestfs are now available:

http://libguestfs.org/download/binaries/debian-packages/

Please read the README file first.

Update

$ time guestfish -a /dev/null run : supported
      augeas yes
     inotify yes
 linuxfsuuid yes
linuxmodules yes
 linuxxattrs yes
        luks yes
        lvm2 yes
       mknod yes
      ntfs3g yes
   ntfsprogs yes
    realpath yes
       scrub yes
     selinux yes
          xz yes
    zerofree yes

real	0m17.335s
user	0m16.061s
sys	0m0.608s

Note that time is running Debian inside a virtual machine. On native hardware with KVM that command should run much more quickly.

libguestfs binaries for Debian and Ubuntu

Over the weekend I rewrote the way supermin appliance building works to make it much more lightweight and mostly distro-independent. As a result, libguestfs now builds from source reasonably simply on both Debian and Ubuntu.

I’m also able to make binaries available for Debian and Ubuntu (10.10), but you must read the README file first.

If these do/don’t work for you, please let me know.