Tip: Edit grub kernel command line in RHEL 7 or CentOS 7

Easy with virt-customize. In this example I’m adding the nosmt option to the command line:

$ virt-customize -a rhel7.img \
    --edit '/etc/default/grub:
      s/^GRUB_CMDLINE_LINUX="/GRUB_CMDLINE_LINUX="nosmt /' \
    --run-command 'grub2-mkconfig -o /boot/grub2/grub.cfg'

nbdkit + xz + curl

I’ve submitted a talk about nbdkit, our flexible, pluggable NBD server, to FOSDEM next year about how you can use nbdkit as a replacement for loopback mounts (or “loop mounts” as I was told off for not calling them last week). In preparation for that talk I ran through it in private to a small Red Hat audience on Monday. If I can I will release that video some time, but I may have to edit out Red Hat “super-secret” stuff first (or most likely not because there aren’t any secrets in it, but I’m still waiting for the internal video to be released).

Anyway this attracted a lot of interest and one question that was asked was why the xz plugin which lets you transparently open and uncompress XZ files on the fly was a plugin at all. Surely it would make more sense for it to be a filter? So it could be used not just to uncompress local files, but also xz-compressed cloud images over HTTPS.

The answer is yes it would! So I fixed it. XZ is now a filter (the plugin is left around but we’ll deprecate it eventually).

You can use it on top of the file plugin, curl plugin or other plugins:

$ nbdkit --filter=xz file file.xz
$ nbdkit --filter=xz curl https://download.fedoraproject.org/pub/fedora/linux/releases/29/Cloud/x86_64/images/Fedora-Cloud-Base-29-1.2.x86_64.raw.xz

This is fun and you can use this to boot the cloud image entirely remotely:

$ qemu-system-x86_64 -machine accel=kvm:tcg \
    -cpu host -m 2048 \
    -drive file=nbd:localhost:10809,if=virtio

However it’s incredibly slow. One problem is that the Fedora mirror sites aren’t very happy about you issuing lots of small HTTP Range requests and I observed that they throttle the connection quite aggressively. The second problem is that the xz block size for these cloud images is too large.

The XZ format (or rather, LZMA format) is divided into streams and blocks. We don’t normally use streams, and many XZ files use a single block. But it’s possible to tell the xz program to use a smaller than default block size, and in that case the output is divided into indexed blocks. Note the block size applies to the uncompressed input, the compressed blocks will have varying sizes, but the index that is created lets us find the block boundaries easily. When a byte is requested we can use a binary search to take us quickly to the compressed block, uncompress it (and cache it), and answer the request. We will only uncompress at most one block instead of the whole file.

For disk images I normally advocate a 16M block size. The current cloud images use (I think) a 192M block size, so both a huge amount of data has to be read over HTTPS to read one uncompressed byte, plus we have to cache very large blocks in RAM.

As an experiment I recompressed the cloud image using xz --block-size=$((16 * 1024 * 1024)) and hosted it locally, and booting is much quicker (albeit still slow because the cloud image contains cloud-init).

But even better we already ship a variety of disk images compressed with a 16M block size for virt-builder here, and these can be booted directly too:

$ nbdkit -U - --filter=xz curl \
        http://builder.libguestfs.org/fedora-29.xz \
        --run \
    'qemu-system-x86_64 -machine accel=kvm:tcg -cpu host -m 2048 -drive file=$nbd,if=virtio'

… although you can’t log in because they all have locked root accounts (virt-builder normally customizes them after download).

New(ish) in libguestfs 1.27.23 — add firstboot batch files to Windows guests

You’ve been able to do this for a while by hand but now virt-sysprep & virt-customize ≥ 1.27.23 let you easily install firstboot scripts into Windows guests:

$ cat /tmp/test.bat
echo Hello I am a batch file
$ virt-customize -a win7.qcow2 --firstboot /tmp/test.bat

Next time the guest boots, check the log file in C:\Program Files\Red Hat\Firstboot\log.txt

This works well for me in Windows 7 guests. It ought to work in other Windows guests too. So far the only other Windows flavour I tested was W2K3 where the service crashed for some unfathomable reason (I’m not very patient with debugging Windows problems).

So let us know how it goes and we’ll try to fix the bugs as we go along.

Using virt-customize to make custom guests with a single backing file

Yesterday I hinted that virt-customize could be used to make custom guests sharing a single backing file. Here is how you do that.

Firstly download a cloud image, or use virt-builder to create one:

$ virt-builder fedora-20 -o backing.img
[   0.0] Downloading: http://libguestfs.org/download/builder/fedora-20.xz
[   1.0] Planning how to build this image
[   1.0] Uncompressing
[  11.0] Opening the new disk
[  15.0] Setting a random seed
[  15.0] Setting passwords
Setting random password of root to 8obMIvmrWe6CCkAv
[  16.0] Finishing off

Now use qemu-img to create overlays for each guest:

$ qemu-img create -b backing.img -f qcow2 guest1.img
$ qemu-img create -b backing.img -f qcow2 guest2.img

You must leave the backing file untouched. In particular don’t try to customize it, else you’ll corrupt all the guests using that backing file.

Now you can customize each guest overlay:

$ virt-customize -a guest1.img \
    --hostname guest1 --timezone Europe/London \
    --install gcc
[   0.0] Examining the guest ...
[   3.0] Setting a random seed
[   3.0] Setting the hostname: guest1
[   3.0] Setting the timezone: Europe/London
[   3.0] Installing packages: gcc

$ virt-customize -a guest2.img \
  --hostname guest2 --install /usr/bin/soffice
[   0.0] Examining the guest ...
[   5.0] Setting a random seed
[   5.0] Setting the hostname: guest2
[   5.0] Installing packages: /usr/bin/soffice

As expected, each guest overlay uses a different amount of space depending on what has been installed:

$ ls -lh guest?.img
-rw-r--r--. 1 rjones rjones 613M Mar 26 15:49 guest1.img
-rw-r--r--. 1 rjones rjones 924M Mar 26 16:04 guest2.img

New tool: virt-customize

The final big feature of libguestfs 1.26 has arrived. Virt-customize is the customization bits from virt-builder, in a separate program. This lets you take any virtual machine and install packages, edit configuration files, run scripts, set passwords and so on.

One of the most requested features for virt-builder is the ability to customize templates while keeping a shared backing file, and virt-customize lets you do this.

Here’s how to use virt-customize:

$ virt-customize -a fedora-20.img \
    --update --install gcc
[   0.0] Examining the guest ...
[  37.0] Setting a random seed
[  37.0] Updating core packages
[ 238.0] Installing packages: gcc

virt-inspector has a way to list out the packages installed in a virtual machine disk image, and we can use it to show that gcc was installed:

$ virt-inspector -a fedora-20.img |
    xmlstarlet sel -t -c '//application[name="gcc"]'
<application>
        <name>gcc</name>
        <version>4.8.2</version>
        <release>7.fc20</release>
        <arch>x86_64</arch>
</application>