Tip: Convert a Windows DVD ISO to a bootable USB key using guestfish

Situation: You have a Windows DVD (or ISO), but like any sane person in 2013 you don’t have a DVD drive on the computer. You want to convert the Windows DVD into a bootable USB key. There are many recipes for this online, but they all require another Windows machine and of course cannot be automated.

However with guestfish (and the always brilliant SYSLINUX doing most of the heavy lifting), this script will unpack the ISO and turn it into a bootable USB key.

Notes:

1. I am not going to support this script. You will need to read the script, look up the commands in the guestfish man page, and understand what it does. Any requests for help will be deleted unread.
2. You need to edit the USB key device before using the script.
3. You need libguestfs ≥ 1.21 with SYSLINUX support compiled in.

#!/bin/bash -

guestfish <<'EOF'
trace on

add-ro en_microsoft_hyper-v_server_2012_x64_dvd_915600.iso

# NB: The next line MUST be changed to your USB drive.
# ANYTHING ON THIS DRIVE WILL BE OVERWRITTEN WITHOUT WARNING.
add /dev/sdX

run

# Inside the appliance, /dev/sda = DVD, /dev/sdb = USB.
# THESE ARE NOT RELATED TO HOST DISK NAMES.

echo "Partitioning the USB disk ..."
part-init /dev/sdb mbr
part-add /dev/sdb p 63 -1
part-set-mbr-id /dev/sdb 1 0xb
part-set-bootable /dev/sdb 1 true
mkfs vfat /dev/sdb1

echo "Copying the contents of the DVD to the USB key ..."
mkmountpoint /cd
mkmountpoint /usb
mount /dev/sda /cd
mount /dev/sdb1 /usb
# XXX We should add cp-r command XXX
debug sh "cp -rP /sysroot/cd/* /sysroot/usb"
#glob cp-a /cd/* /usb
umount /cd
umount /usb
rmmountpoint /cd
rmmountpoint /usb

echo "Making the USB key bootable using SYSLINUX ..."
syslinux /dev/sdb1
mount /dev/sdb1 /
upload /usr/share/syslinux/chain.c32 /chain.c32
write /syslinux.cfg "DEFAULT windows\n\nLABEL windows\nCOM32 chain.c32\nAPPEND fs ntldr=/bootmgr\n"
umount /dev/sdb1
upload /usr/share/syslinux/mbr.bin /dev/sdb

echo "Finished."

EOF

New in libguestfs: Use SYSLINUX or EXTLINUX to make bootable guests

Although grub support in libguestfs is currently on hold because of an unfortunate situation, the latest libguestfs now supports SYSLINUX and EXTLINUX, which is (let’s be frank about this) a much simpler and more sane bootloader than grub/grub2.

In fact, you can make a bootable Linux guest real easily now. Here’s a script:

#!/usr/bin/perl
# Copyright (C) 2013 Red Hat Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

# This ambitious script creates a complete, bootable guest.

use strict;
use warnings;

use Sys::Guestfs;

my $disk = "syslinux-guest.img";

# Find prerequisites.
my $mbr = "/usr/share/syslinux/mbr.bin";
unless (-f $mbr) {
    $mbr = "/usr/lib/syslinux/mbr.bin";
    unless (-f $mbr) {
        die "$0: mbr.bin (from SYSLINUX) not found\n";
    }
}
print "mbr: $mbr\n";

my $mbr_data;
{
    local $/ = undef;
    open MBR, "$mbr" or die "$mbr: $!";
    $mbr_data = <MBR>;
}
die "invalid mbr.bin" unless length ($mbr_data) == 440;

my $kernel = `ls -1rv /boot/vmlinuz* | head -1`;
chomp $kernel;
unless ($kernel) {
    die "$0: kernel could not be found\n";
}
print "kernel: $kernel\n";

print "writing to: $disk ...\n";

# Create the disk.
unlink "$disk";
open DISK, ">$disk" or die "$disk: $!";
truncate DISK, 100*1024*1024;
close DISK;

my $g = Sys::Guestfs->new ();
$g->add_drive ($disk, format => "raw");
$g->launch ();

unless ($g->feature_available (["syslinux"])) {
    die "$0: 'syslinux' feature not available in this version of libguestfs\n";
}

# Format the disk.
$g->part_disk ("/dev/sda", "mbr");
$g->mkfs ("msdos", "/dev/sda1");
$g->mount ("/dev/sda1", "/");

# Install the kernel.
$g->upload ($kernel, "/vmlinuz");

# Install the SYSLINUX configuration file.
$g->write ("/syslinux.cfg", <<_END);
DEFAULT linux
LABEL linux
  SAY Booting the kernel from /vmlinuz
  KERNEL vmlinuz
  APPEND ro root=/dev/sda1
_END

$g->umount_all ();

# Install the bootloader.
$g->pwrite_device ("/dev/sda", $mbr_data, 0);
$g->syslinux ("/dev/sda1");
$g->part_set_bootable ("/dev/sda", 1, 1);

# Finish off.
$g->shutdown ();

After running the script, you can try booting the minimal “guest” (note it only contains a kernel, not any userspace):

$ qemu-kvm -hda syslinux-guest.img

Paper classifying bugs in Linux filesystems

This is an excellent paper classifying bugs in Linux filesystems. The results seem to be generally applicable to bugs in open source kernel code.

Some thoughts after 2.5 weeks with the Samsung Chromebook

A couple of months ago I bought a Samsung Chromebook to use while travelling around Japan. So how did that work out?

I need to point out first that I was mostly using Fedora 17 (but could dual-boot into ChromeOS for occasional things like G+ Hangouts). Fedora 17 isn’t the latest release, and it is still using the ChromeOS kernel, not a Fedora-compiled kernel. I’ve marked with an asterisk all the points that I think are caused by the ChromeOS kernel rather than by Fedora or the Chromebook itself.

1. * Suspend pretty much doesn’t work. It suspends OK, but can’t resume. This all works fine on ChromeOS however, and I suspect this is just a kernel issue which will/has been fixed in F18.
2. Trackpad sucks quite a bit. For example, it’s very hard to accurately right click. Left clicking often causes the mouse to zoom around the screen. It works better in ChromeOS. Is this an X server issue and/or does ChromeOS use X?
3. * There is some hardware clock problem I couldn’t quite fathom. Either the h/w clock is set to the local time or else Fedora cannot save the timezone, but either way, unless NTP was running (and hence I had wifi etc) it always flipped back to UTC after rebooting. Almost certainly some sort of kernel issue that has/will be fixed in F18.
4. It’s very light weight. Carrying it around everywhere was no effort at all.
5. Battery lasts “forever”. Well, at least 6 or 7 hours which was the longest I needed it for. XFCE Power Manager was predicting 8+ hours, but I don’t know how accurate that is.
6. A touch screen would have been a really nice addition. But not if it meant increasing the price.
7. It’s pretty robustly built.
8. It’s fast enough, with enough storage, for serious development. While it’s not blazingly fast like x86-64, it’s good enough even for libguestfs development (libguestfs being a very large program).

I’m going to categorize this one as a definite success.

KVM Forum Barcelona next week

I am giving a short talk about libguestfs at the Linux Foundation KVM Forum in Barcelona next week (full schedule here).

whenjobs 0.7.0 released

I’ve just released whenjobs 0.7.0 to fix a couple of problems that I was having myself.

Firstly, while it’s a nice feature of whenjobs that I can set variables, sometimes I don’t want jobs to be triggered as a result of setting variables. The new whenjobs --whisper command lets me set variables without reevaluating when-clauses:

$ whenjobs --whisper libguestfs_version=1.17.16

Secondly, my jobs file was getting pretty long and unmaintainable (400 lines), since I added various libguestfs branches and hivex to be managed by whenjobs. So I’ve added a feature where you can split the jobs file into multiple files:

$ ls -l .whenjobs/*.ml
-rw-rw-r--. 1 rjones rjones 3284 Mar 13 19:10 .whenjobs/jobs_hivex.ml
-rw-rw-r--. 1 rjones rjones 4039 Mar 13 19:07 .whenjobs/jobs_libguestfs.ml
-rw-rw-r--. 1 rjones rjones 3766 Mar 13 19:09 .whenjobs/jobs_libguestfs_stable.ml
-rw-rw-r--. 1 rjones rjones   45 Mar 13 19:10 .whenjobs/jobs.ml

I think I described whenjobs as a “cron replacement”. It is a cron replacement for me, for my personal use, but it’s not officially a cron replacement and this project has nothing whatsoever to do with Red Hat or Fedora. The reason I say this is I get some pretty idiotic comments like these.

A work in progress: whenjobs — another cron replacement

whenjobs (git repo) is a cron replacement. From the manual page …

Whenjobs is a powerful but simple replacement for cron. It lets you run jobs periodically like cron, but it also lets you trigger jobs to run when user-defined variables are set or change value.

Periodic jobs are written like this:

every 10 minutes :
<<
  # Get the current load average.
  load=`awk '{print $1}' /proc/loadavg`
  whenjobs --set load $load --type float
>>

When-statements let you create jobs that run based on variables set elsewhere:

when load >= 6 :
<<
  mail -s "ALERT: high load average: $load" $LOGNAME < /dev/null
>>

(When statements are "edge-triggered", meaning that this job will only run when the load goes from under 6 to ≥ 6).

The motivation is building things from git automatically. Here is another job script:

Every 10 minutes, get the latest tagged version from the git repository. The variable ‘version’ will be set to something like “v1.2.3″, “v1.2.4″, etc over time as new releases get tagged.

every 10 minutes :
<<
  cd /my/git/repo
  tag=`git-describe --tags`
  whenjobs --set version $tag
>>

When the ‘version’ variable changes (ie. a new release is tagged) try to build it. ‘changes’ is a function that compares the previous value of a variable from when this job last ran with the current value of a variable, and returns true if the previous and current values are different.

when changes version :
<<
  cd /my/git/buildrepo
  git pull
  git reset --hard $version
  ./configure
  make clean all check dist
  whenjobs --set successful_local_build $version
>>

In parallel, build on a remote machine.

when changes version :
<<
  ssh remote ./do_build $version
  whenjobs --set successful_remote_build $version
>>

Only when the new release has been successfully built on local and remote, upload it to the website.

when successful_local_build == version &&
     successful_remote_build == version :
<<
  cd /my/git/buildrepo
  curl -T name-$success.tar.gz ftp://ftp.example.com/upload/
>>

udev unexpectedness

This was unexpected:

Write something to a partition device (eg. /dev/vda1) and immediately call blockdev --rereadpt /dev/vda to re-read the partition table of the whole device. Sometimes (about 50% for me) the blockdev command fails with:

blockdev: BLKRRPART: Device or resource busy

Nothing else is using /dev/vda, nothing from it was mounted, and the error was intermittent which indicates a race condition.

Why this happens:

udev has a rule that runs blkid -o udev -p /dev/vda1. It does this every time you close a block device so that blkid can rescan the content of the device.

The act of blkid running very briefly behind our backs causes the device to be open during the blockdev operation, causing it to fail.

Adding udevadm settle between the close and the blockdev fixed the problem for us, although this command is also inherently racy (what happens if it runs before the kernel has sent a message to udev?)

New in libguestfs — export blkid info

We’ve been using blkid successfully in libguestfs for a while, but in a piecemeal fashion. New in libguestfs ≥ 1.15.9 is that you can now get all the information that blkid knows about any device:

><fs> blkid /dev/vg_f15x32/lv_root
UUID: 9293385f-3200-4694-8f4b-e20bb8d73c37
VERSION: 1.0
TYPE: ext4
USAGE: filesystem
MINIMUM_IO_SIZE: 512
PHYSICAL_SECTOR_SIZE: 512
LOGICAL_SECTOR_SIZE: 512
><fs> blkid /dev/vda1
UUID: d2cd4319-f515-4be2-9a5c-fc8b57b53723
VERSION: 1.0
TYPE: ext4
USAGE: filesystem
MINIMUM_IO_SIZE: 512
PHYSICAL_SECTOR_SIZE: 512
LOGICAL_SECTOR_SIZE: 512
PART_ENTRY_SCHEME: dos
PART_ENTRY_TYPE: 0x83
PART_ENTRY_FLAGS: 0x80
PART_ENTRY_NUMBER: 1
PART_ENTRY_OFFSET: 2048
PART_ENTRY_SIZE: 1024000
PART_ENTRY_DISK: 252:0

Thanks to Wanlong Gao [awesome photo!] for adding this.

Half-baked idea: Programs can be libraries

For more half-baked ideas, see the ideas tag.

Wouldn’t it be great if qemu was a library. Instead of forking qemu in order to run a guest, you’d just create a new “qemu handle” in your program:

/* gcc program.c -o program -lqemu */
qemu_h *guest = qemu_new (/* args */);
qemu_guest_run (guest);
qemu_destroy (guest);

There would be some sort of “librarifier” for ELF binaries that turns them into ELF shared libraries, hiding all the symbols, and creating a basic API that lets you create new instances, pass command line arguments over, and destroy instances.

For bonus points, programs could become “library-aware” and offer more formal APIs.