nbdkit Windows port contd.

We ported nbdkit to Windows. That port is now upstream and should appear in the next stable release (1.24). There is also a new native file plugin for Windows which supports Windows files and volumes, hole punching for sparse files, querying file sparseness, and efficient zeroing.

nbdkit now ported to Windows

This week I ported nbdkit, our high performance plugin-based Network Block Device server, to Windows. Currently it’s not upstream but you can download the Windows branch from here.

There were several possible ways we could have done this including Cygwin which might have been easier, but in the end I did a port to the raw Win32/Winsock APIs. You can compile it on Fedora using the mingw-w64-based Fedora Windows Cross Compiler and run it using Wine. Familiar commands like this work:

$ ./nbdkit.exe -f -v memory 1G

Windows is such a trash pile of awful APIs it’s a wonder how anyone can use it. Like what, and huh and WTF having to split a 64 bit int across two fields in a struct?? Not to mention the whole mess which is HANDLEs vs SOCKETs vs file descriptors and errno handling in Winsock. But I got there in the end.

I got many existing plugins and filters compiled. Not all of those listed will be working, but the main features are fine. You can also write your own plugins to the same API as Linux ones. I’m hoping that someone can write a Windows block device plugin (especially one which integrates with features like VSS).

$ find \( -name '*.exe' -o -name '*.dll' \) -a -printf "%f\n" | sort -u
nbdkit-blocksize-filter.dll
nbdkit-cacheextents-filter.dll
nbdkit-cow-filter.dll
nbdkit-data-plugin.dll
nbdkit-ddrescue-filter.dll
nbdkit-delay-filter.dll
nbdkit-error-filter.dll
nbdkit-example1-plugin.dll
nbdkit.exe
nbdkit-exitlast-filter.dll
nbdkit-extentlist-filter.dll
nbdkit-file-plugin.dll
nbdkit-fua-filter.dll
nbdkit-full-plugin.dll
nbdkit-gzip-filter.dll
nbdkit-gzip-plugin.dll
nbdkit-info-plugin.dll
nbdkit-ip-filter.dll
nbdkit-limit-filter.dll
nbdkit-memory-plugin.dll
nbdkit-nocache-filter.dll
nbdkit-noextents-filter.dll
nbdkit-nofilter-filter.dll
nbdkit-noparallel-filter.dll
nbdkit-nozero-filter.dll
nbdkit-null-plugin.dll
nbdkit-offset-filter.dll
nbdkit-partition-filter.dll
nbdkit-partitioning-plugin.dll
nbdkit-pattern-plugin.dll
nbdkit-random-plugin.dll
nbdkit-rate-filter.dll
nbdkit-readahead-filter.dll
nbdkit-retry-filter.dll
nbdkit-split-plugin.dll
nbdkit-stats-filter.dll
nbdkit-swab-filter.dll
nbdkit-tls-fallback-filter.dll
nbdkit-truncate-filter.dll
nbdkit-zero-plugin.dll

Creating Windows templates for virt-builder

virt-builder is a tool for rapidly creating customized Linux images. Recently I’ve added support for Windows although for rather obvious licensing reasons we cannot distribute the Windows templates which would be needed to provide Windows support for everyone. However you can build your own Windows templates as described here and then:

$ virt-builder -l | grep windows
windows-10.0-server      x86_64     Windows Server 2016 (x86_64)
windows-6.2-server       x86_64     Windows Server 2012 (x86_64)
windows-6.3-server       x86_64     Windows Server 2012 R2 (x86_64)
$ virt-builder windows-6.3-server
[   0.6] Downloading: http://xx/builder/windows-6.3-server.xz
[   5.1] Planning how to build this image
[   5.1] Uncompressing
[  60.1] Opening the new disk
[  77.6] Setting a random seed
virt-builder: warning: random seed could not be set for this type of guest
virt-builder: warning: passwords could not be set for this type of guest
[  77.6] Finishing off
                   Output file: windows-6.3-server.img
                   Output size: 10.0G
                 Output format: raw
            Total usable space: 9.7G
                    Free space: 3.5G (36%)

To build a Windows template repository you will need the latest libguestfs sources checked out from https://github.com/libguestfs/libguestfs and you will also need a suitable Windows Volume License, KMS or MSDN developer subscription. Also the final Windows templates are at least ten times larger than Linux templates, so virt-builder operations take correspondingly longer and use lots more disk space.

First download install ISOs for the Windows guests you want to use.

After cloning the latest libguestfs sources, go into the builder/templates subdirectory. Edit the top of the make-template.ml script to set the path which contains the Windows ISOs. You will also possibly need to edit the names of the ISOs later in the script.

Build a template, eg:

$ ../../run ./make-template.ml windows 2k12 x86_64

You’ll need to read the script to understand what the arguments do. The script will ask you for the product key, where you should enter the volume license key or your MSDN key.

Each time you run the script successfully you’ll end up with two files called something like:

windows-6.2-server.xz
windows-6.2-server.index-fragment

The version numbers are Windows internal version numbers.

After you’ve created templates for all the Windows guest types you need, copy them to any (private) web server, and concatenate all the index fragments into the final index file:

$ cat *.index-fragment > index

Finally create a virt-builder repo file pointing to this index file:

# cat /etc/virt-builder/repos.d/windows.conf
[windows]
uri=http://xx/builder/index

You can now create Windows guests in virt-builder. However note they are not sysprepped. We can’t do this because it requires some Windows tooling. So while these guests are good for small tests and similar, they’re not suitable for creating actual Windows long-lived VMs. To do that you will need to add a sysprep.exe step somewhere in the template creation process.

Tip: virt-install Windows with virtio device drivers

You have to unset these variables because of a long-standing bug in SPICE:

# unset http_proxy
# unset https_proxy

You can’t use virt-install’s --cdrom option twice, because virt-install ignores the second use of the option and only adds a single CD-ROM to the guest. Instead, use --disk ...,device=cdrom,bus=ide:

# virt-install --name=w81-virtio --ram=4096 \
    --cpu=host --vcpus=2 \
    --os-type=windows --os-variant=win8.1 \
    --disk /dev/VG/w81-virtio,bus=virtio \
    --disk en-gb_windows_8.1_pro_n_vl_with_update_x64_dvd_6050975.iso,device=cdrom,bus=ide \
    --disk /usr/share/virtio-win/virtio-win.iso,device=cdrom,bus=ide

During the install you’ll have to select the “Load driver” option and load the right viostor driver from the second CD-ROM (E:).

New(ish) in libguestfs 1.27.23 — add firstboot batch files to Windows guests

You’ve been able to do this for a while by hand but now virt-sysprep & virt-customize ≥ 1.27.23 let you easily install firstboot scripts into Windows guests:

$ cat /tmp/test.bat
echo Hello I am a batch file
$ virt-customize -a win7.qcow2 --firstboot /tmp/test.bat

Next time the guest boots, check the log file in C:\Program Files\Red Hat\Firstboot\log.txt

This works well for me in Windows 7 guests. It ought to work in other Windows guests too. So far the only other Windows flavour I tested was W2K3 where the service crashed for some unfathomable reason (I’m not very patient with debugging Windows problems).

So let us know how it goes and we’ll try to fix the bugs as we go along.

virt-log now supports the Windows Event Log

New virt tool virt-log now supports the Windows Event Log. If you have a recent Windows guest you can display the System event log by doing:

$ virt-log -d Win8 | less

What you will see is a very long XML file.

This requires an Evtx parser. I have now chosen this library for Fedora (it needs a reviewer, as you can see). The code is sensible and maintained.

It also only works for Windows ≥ Vista, because Microsoft completely rewrote the way that log files are stored, from one strange binary format to another strange binary format [so a little different from the systemd journal …].

As usual, patches to virt-log to support other guest operating systems are welcome.

Tip: Convert a Windows DVD ISO to a bootable USB key using guestfish

Situation: You have a Windows DVD (or ISO), but like any sane person in 2013 you don’t have a DVD drive on the computer. You want to convert the Windows DVD into a bootable USB key. There are many recipes for this online, but they all require another Windows machine and of course cannot be automated.

However with guestfish (and the always brilliant SYSLINUX doing most of the heavy lifting), this script will unpack the ISO and turn it into a bootable USB key.

Notes:

1. I am not going to support this script. You will need to read the script, look up the commands in the guestfish man page, and understand what it does. Any requests for help will be deleted unread.
2. You need to edit the USB key device before using the script.
3. You need libguestfs ≥ 1.21 with SYSLINUX support compiled in.

#!/bin/bash -

guestfish <<'EOF'
trace on

add-ro en_microsoft_hyper-v_server_2012_x64_dvd_915600.iso

# NB: The next line MUST be changed to your USB drive.
# ANYTHING ON THIS DRIVE WILL BE OVERWRITTEN WITHOUT WARNING.
add /dev/sdX

run

# Inside the appliance, /dev/sda = DVD, /dev/sdb = USB.
# THESE ARE NOT RELATED TO HOST DISK NAMES.

echo "Partitioning the USB disk ..."
part-init /dev/sdb mbr
part-add /dev/sdb p 63 -1
part-set-mbr-id /dev/sdb 1 0xb
part-set-bootable /dev/sdb 1 true
mkfs vfat /dev/sdb1

echo "Copying the contents of the DVD to the USB key ..."
mkmountpoint /cd
mkmountpoint /usb
mount /dev/sda /cd
mount /dev/sdb1 /usb
# XXX We should add cp-r command XXX
debug sh "cp -rP /sysroot/cd/* /sysroot/usb"
#glob cp-a /cd/* /usb
umount /cd
umount /usb
rmmountpoint /cd
rmmountpoint /usb

echo "Making the USB key bootable using SYSLINUX ..."
syslinux /dev/sdb1
mount /dev/sdb1 /
upload /usr/share/syslinux/chain.c32 /chain.c32
write /syslinux.cfg "DEFAULT windows\n\nLABEL windows\nCOM32 chain.c32\nAPPEND fs ntldr=/bootmgr\n"
umount /dev/sdb1
upload /usr/share/syslinux/mbr.bin /dev/sdb

echo "Finished."

EOF

A quick note about 64 bit Fedora Windows cross-compiler

I don’t often talk about the Fedora Windows cross-compiler project which I guess I helped to start back in 2008.

However last week some legal uncertainty about the 64 bit compiler was lifted and Erik van Pienbroek and Kalev Lember have been doing an amazing job getting a full 32 + 64 bit cross-compiler chain into Fedora 17.

The upshot of this is incredible: From a single code-base, you can build, on Fedora, a Linux binary, 32- and 64- bit Windows binaries, and (with a tiny bit of proprietary code) Mac OS X binaries. This includes cross-platform graphical programs (using Gtk), and over a hundred libraries for graphics, sound, games, networking, databases and much more. It’s not just plain binaries either, mingw-w64 includes a credible DDK replacement so you can build Windows device drivers too.

Android is the new Windows …

You want to know if Android or iPhone is going to “win” in the end? It’s obviously going to be Android, because it’s the new Windows.

Exhibit (a): My Google Nexus S phone: requires rebooting every 3 days otherwise it just stops being able to receive/send calls and/or texts. Data over 3g still works during these outages.

Exhibit (b): My new Samsung Galaxy tab, a Christmas present of sorts from Red Hat: Fresh out of the box(!), this cannot connect to my wifi (absolutely everything else can connect fine to this wifi). Support forums are full of people reporting the same thing, and everyone suggesting you reboot/reinstall/press some magic key sequence/etc. Where have I heard that story before?

libguestfs 1.12.6 for Debian

Thanks to the tireless work of Hilko Bengen, libguestfs 1.12.6 is now available as an official Debian package.

Also, you can compile hivex on Mac OS X and Windows, thanks to Alex Nelson and Gillen Daniel respectively.