libguestfs architecture ASCII art

There was a little bit of confusion about how exactly libguestfs works, so I drew the architecture diagram in wonderful ASCII art … This is what happens when you type guestfish -a disk.img …

  ___________________
 /                   \
 | main program      |
 |                   |
 |                   |           child process / appliance
 |                   |           __________________________
 |                   |          / qemu                     \
 +-------------------+   RPC    |      +-----------------+ |
 | libguestfs     <--------------------> guestfsd        | |
 |                   |          |      +-----------------+ |
 \___________________/          |      | Linux kernel    | |
                                |      +--^--------------+ |
                                \_________|________________/
                                          |
                                   _______v______
                                  /              \
                                  | Device or    |
                                  | disk image   |
                                  \______________/

See here for all the information about the API and architecture.

New tool: virt-list-filesystems

$ virt-list-filesystems Debian5x64.img
/dev/debian5x64/home
/dev/debian5x64/root
/dev/debian5x64/tmp
/dev/debian5x64/usr
/dev/debian5x64/var
/dev/sda1

You can also augment this tool with the -a and -l options. The -a option tells it to list swap partitions too. The -l option tells it to show the filesystem type on each partition that was found:

$ virt-list-filesystems -a -l Fedora12.img
/dev/sda1 ext4
/dev/vg_f12x64/lv_root ext4
/dev/vg_f12x64/lv_swap swap

While this is a fairly simple tool, the use case comes from a user who asked me how I knew what filesystems could be mounted using the guestmount command. The answer is that you don’t know, unless you know something about the guest, or you interactively examine the guest using guestfish, or just use this new tool.

Prebuilt distributions part 2

In part 1 I discussed how these days Linux Live CDs usually come with a prebuilt disk image of the distro which is simply copied over to the hard disk during installation. (The “old” method was to rpm/dpkg-install the packages which is much more time-consuming). However my first test wasn’t very successful because I was using the “cp” command to copy files.

Anaconda (the Fedora installer) is smarter than this. It “dd”s the prebuilt disk image to the hard disk and then uses an ext2/3/4 utility called resize2fs to expand it to the correct size.

I changed the previous guestfish script to take this approach.

The new/Anaconda approach is much faster. Our total time is down from over 18 minutes to 2½ minutes (approximately 2 minutes for the “dd”, 2 seconds for the resize2fs, and the rest of the time taken doing the partitioning and LVM creation).

Unfortunately we have to leave Ubuntu behind at this point. Ubuntu ships with a squashfs, and I’m not aware of any way to turn this into an ext3 partition efficiently (except to use “cp” which we showed in part 1 was very slow). The new script only works with Fedora Live CD ISOs.

The new script is after the cut.

Prebuilt distributions part 1

Previously I took a look at unpacking Fedora and Ubuntu live CDs to find out what’s inside them and to ask the question can we use the prebuilt filesystem image that these live CDs contain to quickly create a Fedora or Ubuntu “all-defaults” virtual machine?

This is my first attempt, and it’s not successful, but it does demonstrate a large and interesting guestfish script doing a non-trivial amount of work.

This script:

1. mounts the prebuilt filesystem from either a Fedora or Ubuntu live CD
2. creates a disk image with a 200 MB /boot partition and a single / (root) logical volume covering the remainder of the disk
3. uses the cp -a command to recursively copy the prebuilt filesystem to the disk

Where it fails is that “cp” isn’t very fast. On my local machine it took 18 minutes to copy all the files across, which means this isn’t a practical “instant install” method. (I didn’t in the end try to boot the final disk image).

In part 2 this week, I’ll look at the approach that anaconda takes: It dd’s the disk image and then runs resize2fs on it to expand it into the available space.

In part 3 I’ll compare this approach to others: virt-install, manual installation, kickstart, cobbler, debootstrap and ubuntu-vm-builder.

The script itself follows after the cut:

The Doomsday Algorithm

Work out, in your head, the day of the week corresponding to any date — eg. today is 2009-11-22, a Sunday — using this method. (Wikipedia page, but I recommend the first link). If you stick to just years from 1900 onwards it’s very simple. Could be a geeky ice-breaker at parties to tell people what day they were born on.

I’m fairly certain I remember a TV one-off in the 1980s where some unfortunate autistic man was subjected to a “Mastermind“-style interrogation. The only thing this poor man could do was tell the day of the week for any date. Probably he’d stumbled upon this method …

Interview: Virtualization improvements in F12

Lots of virt developers, including me, were interviewed for the Fedora 12 release by Mel Chua.

Looking closer at Fedora, Ubuntu live CDs

Previously I’ve shown you can use guestfish to unpack a Fedora live CD.

I’m interested in whether we can use the contents of these live CDs to mass-install operating systems using libguestfs.

If you imagine that you go through an “all defaults” install of say Fedora or Ubuntu to a new virtual machine, then when you end up with is an identical disk image containing 1-2 GB of default packages and a lot of empty space. Two people asked to go through the same all-defaults install of the same distro would end up with roughly the same content. The details on the disk would be slightly different because some parts of the disk partitioning and ext3 superblocks change slightly depending on the overall disk size. But really those things can be fixed up afterwards using a little repartitioning, lvresize and resize2fs.

Let’s look inside an Ubuntu live CD:

$ guestfish --ro -a ubuntu-9.10-desktop-amd64.iso

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help with commands
      'quit' to quit the shell

><fs> run
><fs> list-devices
/dev/sda
><fs> file /dev/sda
ISO 9660 CD-ROM filesystem data 'Ubuntu 9.10 amd64
><fs> mkmountpoint /t1
><fs> mount /dev/sda /t1
><fs> ll /t1/casper
total 691049
dr-xr-xr-x  2 root root      2048 Oct 27 14:31 .
dr-xr-xr-x 10 root root      2048 Oct 27 14:31 ..
-r--r--r--  2 root root     37288 Oct 27 14:19 filesystem.manifest
-r--r--r--  2 root root     35354 Oct 27 14:16 filesystem.manifest-desktop
-r--r--r--  2 root root 697778176 Oct 27 14:25 filesystem.squashfs
-r--r--r--  2 root root   5836401 Oct 27 14:20 initrd.lz
-r--r--r--  2 root root   3941696 Oct 16 12:12 vmlinuz
><fs> mkmountpoint /t2
><fs> mount-loop /t1/casper/filesystem.squashfs /t2
><fs> cat /t2/etc/debian_version
squeeze/sid

The file /casper/filesystem.squashfs seems to be a complete Ubuntu installation, and if I’m understanding this correctly the Ubuntu installer will copy this to the newly created filesystem directly. That will be the new Ubuntu installation, plus or minus some config file changes and some extra packages downloaded afterwards from the net.

Here’s the same examination of the Fedora 12 Live CD:

$ guestfish --ro -a Fedora-12-x86_64-Live.iso

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help with commands
      'quit' to quit the shell

><fs> run
><fs> list-devices
/dev/sda
><fs> file /dev/sda
ISO 9660 CD-ROM filesystem data 'Fedora-12-x86_64-Live
><fs> mkmountpoint /t1
><fs> mount /dev/sda /t1
><fs> ls /t1
EFI
GPL
LiveOS
isolinux
><fs> ll /t1/LiveOS
total 655291
dr-xr-xr-x 2 root root      2048 Nov  9 14:45 .
dr-xr-xr-x 5 root root      2048 Nov  9 14:44 ..
-r-xr-xr-x 1 root root     23040 Nov  9 14:44 livecd-iso-to-disk
-r-xr-xr-x 1 root root      8192 Nov  9 14:45 osmin.img
-r-xr-xr-x 1 root root 670982144 Nov  9 14:48 squashfs.img
><fs> mkmountpoint /t2
><fs> mount-loop /t1/LiveOS/squashfs.img /t2
><fs> ls /t2
LiveOS
><fs> ls /t2/LiveOS/
ext3fs.img
><fs> mkmountpoint /t3
><fs> mount-loop /t2/LiveOS/ext3fs.img /t3
><fs> cat /t3/etc/redhat-release
Fedora release 12 (Constantine)

Again I hope I assume correctly that the installer copies ext3fs.img to the hard disk when installing Fedora 12.

So my vague plan, assuming anything I’ve written above is correct, is to take these pre-made filesystem images and allow people to quickly install specific operating system images from a simple tool:

$ virt-press Fedora-12 F12

which would stamp out a Fedora 12 VM in a few seconds and register it with libvirt as “F12″.

I’m not sure this is possible yet …

Half-baked ideas: View source button for Fedora

For more half-baked ideas, see my ideas tag.

I’ll mention first that this isn’t my idea, and it’s not new or original. OLPC already implemented a View Source button.

Why can’t we have the same for Fedora? This is how it would work …

The View Source button would hover in your task bar. When pressed it opens up this dialog:

Like xkill and xwininfo, pressing the “point at a window” button changes your mouse so you click on the program you want to view the source of. X sort of makes it possible to find out (with a bit of effort) which binary is behind each program (see for example the xprop command).

You do an rpm -qf on this binary (or use a yum search) to locate the source.

Use yumdownloader --source to download the source. Unpack it into a standard rpmbuild location, and open up the user’s preferred editor.

With experience, and many custom rules and heuristics, you can extend this idea. For example, if they pointed at a dialog box, search the source for strings from the dialog box to try to locate the exact lines of code.

Or have some debuginfo-like metadata packages which are generated when packages are built, to allow very precise file/line locations to be determined.

Combine the whole thing with LXR so we can browse source intuitively.

This is a great way to encourage contributions to Fedora and Free software in general, because I think it would really make code much more accessible to casual programmers, tinkerers and children. Even experienced programmers would find it useful when tracking down bugs in random applications.

Can we disable Firefox’s stupid self-signed encryption dialog?

A lot has been written about how Firefox’s stupid dialog is a big step backwards for the web.

But is there a way to disable it? Ideally I’d like it to work like ssh – give me a simple single-click warning and display the certificate the first time, and after that don’t say anything at all unless the certificate changes unexpectedly.

Update

This paper on phishing [PDF] is excellent.

Fedora 12 is out! What can you do to help?

Fedora 12 is out, but what can you do to help?

• Join one or two of the most popular torrents (I’m seeding Fedora-12-x86_64-DVD.torrent and Fedora-12-x86_64-Live.torrent).
• Install and test Fedora 12 and write good bug reports. If you don’t want to install Fedora on a new machine, install it in a VM instead!
• Join the Fedora project — we need much more than just programmers: writers, artists, translators and testers are all needed and welcome.