Making a bootable CD-ROM/ISO from virt-builder

virt-builder can throw out new virtual machines with existing operating systems in a few seconds, and you can also write these directly to a USB key or hard disk:

# virt-builder fedora-20 -o /dev/sdX

What you’ve not been able to do is create a bootable CD-ROM or ISO image.

For that I was using the awful livecd-creator program. This needs root and is incredibly fragile. You can have a kickstart that works one day, but not the next, and requires massive hacks to get working … which is the exact reason why I set off to find out how to make virt-builder create ISOs.

Read-only

The background as to why this is difficult: CDs are not writable.

You can take all the files from a Fedora guest built by virt-builder and turn them into an ISO, and put ISOLINUX on it but such a guest would not be able to boot, or at least, it would fail the first time it tried to write to the disk. One day overlayfs (which just went upstream a few days ago) will solve this, but until that is widely available in upstream kernels, we’re going to need something that creates a writable overlay at boot time.

Boot Time

I have chosen dracut (another tool I have a love/hate, mainly hate, relationship with), which has a useful module called dmsquash-live. This implements the boot side of making a live CD writable, for Fedora and RHEL. It’s what livecd-creator uses.

dmsquash-live demands a very particular ISO layout, but it wasn’t hard to reverse engineer it by reading the code carefully and a lot of trial and error.

It requires that we have a filesystem containing a squashfs in a particular location on the CD:

/LiveOS/squashfs.img

That squashfs has to contain inside it a disk image with this precise name:

/LiveOS/rootfs.img

and the disk image is the root filesystem.

The Script

The script below creates all of this, and effectively replaces livecd-creator with something manageable that doesn’t require root, and is only 100 lines of shell (take that OO/Python!)

Update: Kashyap notes that the script will fail if you’re using tmp-on-tmpfs, so you might need to disable that or modify the script to use /var/tmp instead.

Once you’ve run the script you can try booting the image using:

$ qemu-kvm -m 2048 -cdrom boot.iso -boot d

The Future

One improvement to this script would be to remove the dependency on dmsquash-live. We don’t need the baroque complexity of this script, and could write a custom dracut module (perhaps even, a tiny self-contained initramfs) which would do what we need. It could even use overlayfs to simplify things greatly.

#!/bin/bash -

set -e

# Make bootable ISO from virt-builder
# image.
#
# This requires the Fedora
# squashfs/rootfs machinery.  See:
# /lib/dracut/modules.d/90dmsquash-live/dmsquash-live-root.sh

cd /tmp

# Build the regular disk image, but also
# build a special initramfs which has
# the dmsquash-live & pollcdrom modules
# enabled.  We also need to kill SELinux
# relabelling, and hence SELinux.
cat > postinstall <<'EOF'
#!/bin/bash -
version=` rpm -q kernel | sort -rV | head -1 | sed 's/kernel-//' `
echo installed kernel version: $version
dracut --no-hostonly --add "dmsquash-live pollcdrom" /boot/initrd0 $version
EOF

virt-builder fedora-20 \
    --install kernel \
    --root-password password:123456 \
    --edit '/etc/selinux/config:
        s/SELINUX=enforcing/SELINUX=disabled/' \
    --delete /.autorelabel \
    --run postinstall

# Extract the root filesystem (as an ext3/4 disk image).
guestfish --progress-bars --ro -a fedora-20.img -i \
    download /dev/sda3 rootfs.img

# Update /etc/fstab in the rootfs (but NOT in the original guest)
# so it works for the CD
virt-customize -a rootfs.img \
  --write '/etc/fstab:/dev/root / ext4 defaults 1 1'

# Turn the rootfs.img into a squashfs
# which must contain the layout
# /LiveOS/rootfs.img
rm -rf CDroot
rm -f squashfs.img
mkdir -p CDroot/LiveOS
mv rootfs.img CDroot/LiveOS
mksquashfs CDroot squashfs.img

# Create the CD layout.
rm -rf CDroot
mkdir -p CDroot/LiveOS

cp squashfs.img CDroot/LiveOS/

mkdir CDroot/isolinux

# Get the kernel (only) from the disk
# image.
pushd CDroot/isolinux
virt-builder --get-kernel ../../fedora-20.img
mv vmlinuz* vmlinuz0
rm init*
popd

# Get the special initrd that we built
# above.
guestfish --ro -a fedora-20.img -i \
    download /boot/initrd0 CDroot/isolinux/initrd0

# ISOLINUX configuration.
cat > CDroot/isolinux/isolinux.cfg <<EOF
prompt 1
default 1
label 1
    kernel vmlinuz0
    append initrd=initrd0 rd.live.image root=CDLABEL=boot rootfstype=auto rd.live.debug console=tty0 rd_NO_PLYMOUTH
EOF

# Rest of ISOLINUX installation.
cp /usr/share/syslinux/isolinux.bin CDroot/isolinux/
cp /usr/share/syslinux/ldlinux.c32 CDroot/isolinux/
cp /usr/share/syslinux/libcom32.c32 CDroot/isolinux/
cp /usr/share/syslinux/libutil.c32 CDroot/isolinux/
cp /usr/share/syslinux/vesamenu.c32 CDroot/isolinux/

# Create the ISO.
rm -f boot.iso
mkisofs -o boot.iso \
    -J -r \
    -V boot \
   -b isolinux/isolinux.bin -c isolinux/boot.cat \
   -no-emul-boot -boot-load-size 4 -boot-info-table \
   CDroot

2 Comments

Filed under Uncategorized

libguestfs 1.28 released

The new stable version of libguestfs — a C library and tools for accessing and modifying virtual machine disk images — has been released.

There is one brand new tool, virt-log. And I rewrote the virt-v2v and virt-p2v tools. These tools convert VMware and Xen guests and physical machines, to run on KVM. They are now much faster and better than before.

As well as that there are hundreds of other improvements and bug fixes. For a full list, see the release notes.

Libguestfs 1.28 will be available shortly in Fedora 21, Debian/experimental, RHEL and CentOS 7, and elsewhere.

1 Comment

Filed under Uncategorized

Streaming NBD server

The command:

qemu-img convert input output

does not work if the output is a pipe.

It’d sure be nice if it did though! For one thing, we could use this in virt-v2v to stream images into OpenStack Glance (instead of having to spool them into a temporary file).

I mentioned this to Paolo Bonzini yesterday and he suggested a simple workaround. Just replace the output with:

qemu-img convert -n input nbd:...

and write an NBD server that turns the sequence of writes from qemu-img into a stream that gets written to a pipe. Assuming the output is raw, then qemu-img convert will write, starting at disk offset 0, linearly through to the end of the disk image.

How to write such an NBD server easily? nbdkit is a project I started to make it easy to write NBD servers.

So I wrote a streaming plugin which does exactly that, in 243 lines of code.

Using a feature called captive nbdkit, you can rewrite the above command as:

nbdkit -U - streaming pipe=/tmp/output --run '
  qemu-img convert -n input -O raw $nbd
'

(This command will “hang” when you run it — you have to attach some process to read from the pipe, eg: md5sum < /tmp/output)

Further work

The streaming plugin will a lot more generally useful if it supported a sliding window, allowing limited reverse seeking and reading. So there’s a nice little project for a motivated person. See here

2 Comments

Filed under Uncategorized

Tip: Set a root password on a disk image

virt-sysprep --enable customize --root-password password:123456 -a /dev/sdX

Useful when installing Fedora on ARM machines that only have a serial port.

Leave a comment

Filed under Uncategorized

Odd/scary RHEL 5 bug

Yesterday my colleague gave me a RHEL 5 VM disk image which failed to boot after converting it using the latest virt-v2v.  Because it booted before conversion but not afterwards, the fingers naturally pointed at something that we were doing during the conversion process. Which is not unusual as v2v conversion is highly complex.

Screenshot_xen-pv-rhel5.8-x86_64
The “GRUB _” prompt after conversion

The thing is that we don’t reinstall grub during conversion, but we do edit a few grub configuration files. Could editing grub configuration cause this error?

I wanted to understand what the grub-legacy “GRUB _” prompt means. There are lots and lots and lots of people reporting this bug (eg), but as is often the case I could find no coherent explanation anywhere of what grub-legacy means when it gets into this state. Lots of the blind leading the blind, and random suggestions about how people had rescued such machines (probably coincidentally), but no hard data anywhere. So I had to go back to first principles and debug qemu to find out what’s happening just before the message is printed.

Tip: To breakpoint qemu when the Master Boot Record (first sector) is loaded, do:

target remote tcp::1234
set architecture i8086
b *0x7c00
cont

After an evening of debugging, I found that it’s the first sector (known in grub-legacy as “stage 1″) which prints the GRUB<space> message. (The same happens to be true of grub2). The stage 1 boot sector has, written into it at a fixed offset, the location of the /boot/grub/stage2 file, ie. the literal disk start sector and length of this file. It sends BIOS int $0x13 commands to load those sectors into memory at address 0x8000, and jumps there to start the stage 2 of grub. The boot sector is 512 bytes, so there’s no luxury to do anything except print 5 characters. It’s after the stage2 file has been loaded when all the nice graphical stuff happens.

Unfortunately in the image after conversion, the stage2 data loaded into memory was all zeroes, and that’s why the boot fails and you see GRUB<space><cursor> and then the VM crashes.

The mystery was how conversion could be changing the location of the /boot/grub/stage2 file so that it could no longer be loaded at the fixed offset encoded in the boot sector.

This morning it dawned on me what was really happening …

The new virt-v2v tries very hard to avoid copying any unused data from the guest, just to save time. No point wasting time copying deleted files and empty space. This makes virt-v2v very fast, but it has an unusual side-effect: If a file is deleted on the source, the contents of the file are not copied over to the target, and turn into zeroes.

It turns out if you take the source disk image and simply zero all of the empty space in /boot, then the source doesn’t boot either, even though virt-v2v is not involved. Yikes … this could be a bug in RHEL 5. Grub is generating a bootloader that references a deleted file.

This is where we are right now with this bug. It appears that a valid sequence of steps can make a RHEL 5 bootloader that references a deleted file, but still works as long as you never overwrite the sectors used by that file.

I have written a simple test script that you can download to find out if your RHEL ≤ 6 virtual machines could be affected by this problem. I’m interested if anyone else sees this. I ran the test over a selection of RHEL 3 – 5 guests, and could not find any which had the problem, but my collection is not very extensive, and there are likely to be common modes in how they were created.

The next steps will likely be to test a lot more RHEL 5 installs to see if this bug is really common or a strange one-off. I will also probably add a workaround to virt-v2v so it doesn’t trim the boot partition — the reason is that we cannot go back and fix old RHEL 5 installs, we have to work with them if they are broken. If it turns out to be a real bug in RHEL 5 then we will need to issue a fix for that.

2 Comments

Filed under Uncategorized

Fedora 21 Virtualization Test Day is Thursday September 25

https://fedoraproject.org/wiki/Test_Day:2014-09-25_Virtualization

Leave a comment

Filed under Uncategorized

virt-v2v preview packages for RHEL and CentOS 7.1 are available

virt-v2v is a small program for converting guests from VMware or Xen, to run on KVM, RHEV-M or OpenStack. For RHEL 7.1, I am rewriting and enhancing virt-v2v, so it’s much faster and easier to use.

To install, follow the instructions here for setting up the yum repository, and then you can do:

yum install virt-v2v

To use it, start with the manual here that has lots of examples and the full reference documentation.

6 Comments

Filed under Uncategorized