Notes on producing a minimal, compressed filesystem

You can save the minimal, compressed filesystem from a guest in a way that lets you reconstruct that filesystem (or guest) later very quickly. I’m writing an experimental tool called virt-builder which will make this automatic, but here’s how to make the minimal, compressed filesystem manually.

I’m starting here with a freshly installed Fedora 13 i686 guest. Firstly we see what filesystems it contains:

$ virt-list-filesystems -al /dev/vg_pin/TmpF13
/dev/sda1 ext4
/dev/vg_virtbuilder/lv_root ext4
/dev/vg_virtbuilder/lv_swap swap

We have a boot partition (/dev/sda1), a root filesystem (/dev/vg_virtbuilder/lv_root), and a swap partition. We can just ignore swap since it doesn’t contain any information and we can reconstruct it at will.

There is also some extra information “hidden” in the disk and not in a partition, namely the boot sector, partition table, and (maybe) GRUB stages. For this guest just the boot sector is interesting, other guests may have a boot loader located between the boot sector and the first partition. I’m going to ignore these for now, although virt-builder will need to restore this in order to make a bootable guest.

Let’s grab the boot and root partitions using guestfish:

$ guestfish --ro -i -a /dev/vg_pin/TmpF13 \
    download /dev/sda1 fedora-13-i686-boot.img
$ guestfish --progress --ro -i -a /dev/vg_pin/TmpF13 \
    download /dev/vg_virtbuilder/lv_root fedora-13-i686-root.img
 100% ⟦▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉⟧ 00:00

The boot partition is small enough that I’m just going to leave it as it is for now. But I need to shrink the root filesystem. The reason is that I might want to restore this to a different-sized, possibly smaller guest than the one I just created. There is only about 2GB of data in the root filesystem, but as an ext4 filesystem it is taking up about 6GB of space. The resize2fs command has a handy -M option which makes this very simple:

$ e2fsck -f fedora-13-i686-root.img
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Fedora-13-i686-L: 91151/368640 files (0.1% non-contiguous), 531701/1458176 blocks
$ resize2fs -M fedora-13-i686-root.img
resize2fs 1.41.12 (17-May-2010)
Resizing the filesystem on fedora-13-i686-root.img to 520344 (4k) blocks.
The filesystem on fedora-13-i686-root.img is now 520344 blocks long.

Note that new size: 520344 x 4k blocks = 2081376k. We can therefore truncate the file down to the new size without any loss of information:

$ truncate -s 2081376k fedora-13-i686-root.img
$ ls -lh
total 2.5G
-rw-rw-r--. 1 rjones rjones 500M Oct 30 10:21 fedora-13-i686-boot.img
-rw-rw-r--. 1 rjones rjones 2.0G Oct 30 10:30 fedora-13-i686-root.img

I haven’t finished yet. I now need to compress both filesystems so I have something small(ish) and portable. I performed some tests, and xz was the clear winner in terms of final compressed size, although it does take a very long time to run with the -9 option.

$ xz -9 *.img
$ ls -lh
total 652M
-rw-rw-r--. 1 rjones rjones 250M Oct 30 10:21 fedora-13-i686-boot.img.xz
-rw-rw-r--. 1 rjones rjones 403M Oct 30 10:30 fedora-13-i686-root.img.xz

From the two files above, plus the boot sector stuff that I glossed over, it is possible to reconstruct a Fedora 13 VM of any dimensions quickly (in about 1 minute 15 seconds on my machine). Note the comments are not part of the command, and the guest is fully bootable:

guestfish -x -- \
# create a disk image of any size
  alloc test.img 10G : \
  run : \
# create the partitions
  part-init /dev/sda mbr : \
  part-add /dev/sda primary 2048 1026047 : \
  part-add /dev/sda primary 1026048 -64 : \
# upload the boot loader
  upload fedora-13-i686-grub.img /dev/sda : \
# upload the boot partition
  upload <( xzcat fedora-13-i686-boot.img.xz ) /dev/sda1 : \
# create the logical volumes for root and swap
  pvcreate /dev/sda2 : \
  vgcreate vg_virtbuilder /dev/sda2 : \
  lvcreate lv_swap vg_virtbuilder 512 : \
  lvcreate lv_root vg_virtbuilder 256 : \
  lvresize-free /dev/vg_virtbuilder/lv_root 100 : \
# make fresh swap
  mkswap /dev/vg_virtbuilder/lv_swap : \
# upload and resize the root filesystem
  upload <( xzcat fedora-13-i686-root.img.xz ) /dev/vg_virtbuilder/lv_root : \
  resize2fs /dev/vg_virtbuilder/lv_root

Create a partitioned device from a collection of filesystems

Xen has a feature where it can export virtual partitions directly to virtual machines. You can configure a Xen VM like this example:

disk = ['phy:raidvg/devroot,hda1,w','phy:raidvg/devswap,hda2,w']

Notice that host device /dev/raidvg/devroot is mapped to a partition inside the guest (/dev/hda1), and on the host this device directly contains a filesystem:

host# file - < /dev/raidvg/devroot
/dev/stdin: Linux rev 1.0 ext3 filesystem data, UUID=... (needs journal recovery)

Inside the guest, it sees /dev/hda1, /dev/hda2, but no /dev/hda device or partition table.

This is actually a nice feature of Xen because resizing filesystems directly is much easier than resizing a partitioned block device. You can just make the host device bigger (lvresize -L sizeG /dev/raidvg/devroot), reboot the guest so it sees the increased device size, then resize the filesystem (resize2fs — this can even be done live if you want to make the filesystem bigger).

Imagine if we’d been dealing with a KVM partitioned block device instead:

+-+---------------------+------------+
|M| hda1                | hda2       |
|B| (root filesystem)   | (swap)     |
|R|                     |            |
+-+---------------------+------------+

Resizing this is much more painful. You first have to extend the host block device:

+-+---------------------+------------+-------+
|M| hda1                | hda2       | space |
|B| (root filesystem)   | (swap)     |       |
|R|                     |            |       |
+-+---------------------+------------+-------+

Now what do you do? Easiest is probably to create a third (hda3) partition in that extra space. If you didn’t have the foresight to use LVM, then this means your root filesystem cannot be extended — you can only create another extra filesystem (say for /var) and copy files over. This is very inflexible.

Instead you could recalculate the MBR and move (ie. copy block by block) hda2 up. (Imagine it wasn’t swap space since you can just throw that away and recreate it, but some valuable files). Recalculating the MBR is generally error-prone because partitions have strange limitations and alignment requirements.

One day I intend to write a program to do these kinds of complex resizing operations …

Anyhow, this wasn’t even what this rambling blog entry was about. It is a companion to last week’s tip about extracting filesystems from disk images. Can we do the opposite, ie. create a partitioned device from a collection of Xen filesystems?

Answer, yes we can, with guestfish.

I’m starting in fact with the filesystem and swap devices copied from my Xen server, and I need to know their exact sizes in 1024-byte-blocks first:

$ ls --block-size=1024 -l devroot devswap
-rw-rw-r--. 1 rjones rjones 3145728 2010-03-17 14:18 devroot
-rw-rw-r--. 1 rjones rjones 1048576 2010-03-17 14:19 devswap

I’m going to put this into a 5G disk image, giving me space to expand the root filesystem to fit. Inexplicably I’ve decided to keep the swap partition content even though in reality I would just throw it away and recreate the swap partition (imagine there’s some important filesystem content in there instead). I want devswap to precisely fit at the end of the new disk image.

Let’s create the disk image and find out how big it is in sectors:

$ rm -f disk.img
$ truncate -s 5G disk.img
$ guestfish -a disk.img -a devroot -a devswap
><fs> run
><fs> blockdev-getsz /dev/vda
10485760  # size in 512 byte sectors

Now I need to do some back of the envelope calculations to work out how I will size and place each partition. (This is a huge pain in the neck — I had to do several runs to get the numbers to come out right …)

><fs> part-init /dev/vda mbr
# numbers below are in units of 512 byte sectors:
><fs> part-add /dev/vda primary 64 8388607
><fs> part-add /dev/vda primary 8388608 -1
><fs> sfdisk-l /dev/vda

Disk /dev/vda: 10402 cylinders, 16 heads, 63 sectors/track
Units = cylinders of 516096 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/vda1          0+   8322-   8322-   4194272   83  Linux
/dev/vda2       8322+  10402-   2081-   1048576   83  Linux
/dev/vda3          0       -       0          0    0  Empty
/dev/vda4          0       -       0          0    0  Empty

Notice the number of (1024-byte) blocks for devswap is exactly the correct size: 1048576.

The sfdisk-l command is also telling me that my partitions aren’t aligned on “cylinders” which I don’t care about. But the swap partition should be aligned for the underlying device because sector 8388608 == 8192 * 1024.

Once the hard bit is out of the way, I can now copy across my filesystems. Notice I added devroot and devswap as devices (the -a option to guestfish). They appear in the guest as /dev/vdb and /dev/vdc respectively and I can just dd them to the right places:

><fs> dd /dev/vdb /dev/vda1
><fs> dd /dev/vdc /dev/vda2

and resize the root filesystem to fit the space available:

><fs> e2fsck-f /dev/vda1
><fs> resize2fs /dev/vda1

Now I have a single partitioned device, suitable for use with KVM (mind you, not bootable because it still contains a Xen paravirt kernel):

$ virt-list-filesystems -al disk.img
/dev/sda1 ext3
/dev/sda2 swap

As you can see there is much scope for automation …

Prebuilt distributions part 3

[This is the final article in a three part series. Read part 1 and part 2.]

In this part I was going to discuss the different installation options, and I’ll give a quick introduction to each at the end. But you know? What I really need is this tool:

+--------------------------------------------------------+
| Rich's Amazing Bug Environment Reproducer Tool!        |
+--------------------------------------------------------+
| Bugzilla number: # [545116] [ Go ]                     |
|                                                        |
| Preselecting Fedora Rawhide image.  If this is wrong,  |
| choose the right image from the list below:            |
|                                                        |
| [X] Fedora Rawhide (x86-64)                            |
| [ ] Fedora 12                                          |
| [ ] Fedora 11                                          |
| [ ] Ubuntu 9.11                                        |
| [ ] Ubuntu 9.04                                        |
| [ ] Debian 6.0                                         |
| [ ] Debian 5.0    [Download more distributions]        |
|                                                        |
| [X] x86-64   [ ] i386                                  |
|                                                        |
| [Create  virtual machine now]                          |
+--------------------------------------------------------+

The Create virtual machine now button must take no more than 60 seconds to deliver the virtual machine, plus maybe another 60 seconds to boot it and give me the login prompt. I can then go in, configure it and reproduce the bug, and snapshot it, so I can go back and test fixes later. (I’d like the snapshot workflow to go through the user interface too, and ponies).

The only approach that doesn’t take ages to install is to have the distributions prebuilt as small disk images locally (as described already in part 2).

Note stuff I don’t want to have to specify: storage, download URLs, VM names, etc etc.

Why doesn’t this exist already, or perhaps it does?

I’d also like to suggest that the resizable disk image should become yet another distribution format that Linux distros provide. Many do already, albeit buried inside a live CD.

---

As promised, my quick summary of installation methods:

virt-install

virt-install is a great command line tool for installing new VMs. I have these two commands pretty much permanently in my command history:

# lvcreate -n F13Rawhidex64 -L 10G vg_trick
# virt-install -v -n F13Rawhidex64  --accelerate \
    -r 512 -f /dev/vg_trick/F13Rawhidex64 \
    --cdrom Fedora-12-x86_64-DVD.iso

The commands above lead to a manual installation, but you can also specify a kickstart file, although the syntax for doing this is not obvious and not documented in the manual page.

Nevertheless, even an install from a CD-ROM on the local drive (as in the example above) is a slow process, taking anything up to an hour.

kickstart

Kickstart is the method used to automate Anaconda installs, ie. installs of all Fedora, Red Hat and derived distributions. It’s a little unwieldy because you have to write a separate kickstart file, then host that on a web server (so the VM can find it), then add a special magic kernel parameter when booting the VM: ks=http://local.example.com/kickstart.ks

I found in the past that it’s better to keep a collection of kickstart files around and iterate them slowly so you can find out what works and what doesn’t.

Also, kickstarts are automatic, but just as slow as manual installation.

debootstrap

Debian’s debootstrap (and the Fedora equivalent I wrote called febootstrap) is, as far as I know, the standard way to install a new Debian instance from an existing Debian instance. Unlike our VM installs above, debootstrap creates a “chroot” directory, and what you do with that directory is then up to you. Usually you would debootstrap into some mounted device. debootstrap is very simple to use and driven completely from the command line.

It’s not particularly fast. Because it downloads, unpacks and installs each package, it can take an hour or more to run.

ubuntu-vm-builder

This is Ubuntu’s wrapper around debootstrap and libvirt/qemu.

cobbler

Cobbler is a network install server which lets you pxeboot and configure a network of machines (and virtual machines) from a central server. It has lots of wonderful features — go and read the web site.

Prebuilt distributions part 2

In part 1 I discussed how these days Linux Live CDs usually come with a prebuilt disk image of the distro which is simply copied over to the hard disk during installation. (The “old” method was to rpm/dpkg-install the packages which is much more time-consuming). However my first test wasn’t very successful because I was using the “cp” command to copy files.

Anaconda (the Fedora installer) is smarter than this. It “dd”s the prebuilt disk image to the hard disk and then uses an ext2/3/4 utility called resize2fs to expand it to the correct size.

I changed the previous guestfish script to take this approach.

The new/Anaconda approach is much faster. Our total time is down from over 18 minutes to 2½ minutes (approximately 2 minutes for the “dd”, 2 seconds for the resize2fs, and the rest of the time taken doing the partitioning and LVM creation).

Unfortunately we have to leave Ubuntu behind at this point. Ubuntu ships with a squashfs, and I’m not aware of any way to turn this into an ext3 partition efficiently (except to use “cp” which we showed in part 1 was very slow). The new script only works with Fedora Live CD ISOs.

The new script is after the cut.

Prebuilt distributions part 1

Previously I took a look at unpacking Fedora and Ubuntu live CDs to find out what’s inside them and to ask the question can we use the prebuilt filesystem image that these live CDs contain to quickly create a Fedora or Ubuntu “all-defaults” virtual machine?

This is my first attempt, and it’s not successful, but it does demonstrate a large and interesting guestfish script doing a non-trivial amount of work.

This script:

1. mounts the prebuilt filesystem from either a Fedora or Ubuntu live CD
2. creates a disk image with a 200 MB /boot partition and a single / (root) logical volume covering the remainder of the disk
3. uses the cp -a command to recursively copy the prebuilt filesystem to the disk

Where it fails is that “cp” isn’t very fast. On my local machine it took 18 minutes to copy all the files across, which means this isn’t a practical “instant install” method. (I didn’t in the end try to boot the final disk image).

In part 2 this week, I’ll look at the approach that anaconda takes: It dd’s the disk image and then runs resize2fs on it to expand it into the available space.

In part 3 I’ll compare this approach to others: virt-install, manual installation, kickstart, cobbler, debootstrap and ubuntu-vm-builder.

The script itself follows after the cut: