virt-builder Debian 9 image available

Debian 9 (“Stretch”) was released last week and now it’s available in virt-builder, the fast way to build virtual machine disk images:

$ virt-builder -l | grep debian
debian-6                 x86_64     Debian 6 (Squeeze)
debian-7                 sparc64    Debian 7 (Wheezy) (sparc64)
debian-7                 x86_64     Debian 7 (Wheezy)
debian-8                 x86_64     Debian 8 (Jessie)
debian-9                 x86_64     Debian 9 (stretch)

$ virt-builder debian-9 \
    --root-password password:123456
[   0.5] Downloading: http://libguestfs.org/download/builder/debian-9.xz
[   1.2] Planning how to build this image
[   1.2] Uncompressing
[   5.5] Opening the new disk
[  15.4] Setting a random seed
virt-builder: warning: random seed could not be set for this type of guest
[  15.4] Setting passwords
[  16.7] Finishing off
                   Output file: debian-9.img
                   Output size: 6.0G
                 Output format: raw
            Total usable space: 3.9G
                    Free space: 3.1G (78%)

$ qemu-system-x86_64 \
    -machine accel=kvm:tcg -cpu host -m 2048 \
    -drive file=debian-9.img,format=raw,if=virtio \
    -serial stdio

Raise the Itanic! (part 2)

Previously …

I finally got my HP Integrity RX2620 (dual core Itanium II) working, after a lot of messing around in the EFI shell. It is now running Debian-ia64 Wheezy. The thing is so noisy that you can’t stay in the room with it for very long, and I discovered that if you pull out the fans then it overheats and shuts down after a few minutes.

Here is lstopo output:

and /proc/cpuinfo:

processor  : 0
vendor     : GenuineIntel
arch       : IA-64
family     : 31
model      : 2
model name : Madison up to 9M cache
revision   : 2
archrev    : 0
features   : branchlong
cpu number : 0
cpu regs   : 4
cpu MHz    : 1600.035
itc MHz    : 1600.035576
BogoMIPS   : 2390.01
siblings   : 1
physical id: 0

processor  : 1
vendor     : GenuineIntel
arch       : IA-64
family     : 31
model      : 2
model name : Madison up to 9M cache
revision   : 2
archrev    : 0
features   : branchlong
cpu number : 0
cpu regs   : 4
cpu MHz    : 1600.035
itc MHz    : 1600.035576
BogoMIPS   : 2390.01
siblings   : 1
physical id: 1

Also: Nooooooooooooooooooo!!!!11!!1

MIPS Creator (32 bit dual core MIPS)

As I mentioned in passing yesterday I have a MIPS Creator CI20. It’s a tiny single board dual core 32 bit MIPS computer. The cost was £55 including tax and delivery.

It comes with Debian mipsel pre-installed, so all I had to do was boot it the first time with an HDMI monitor and keyboard, configure the WLAN, disable the autologin, reboot and done: I have a tiny MIPS server.

One tip: The hardware clock was way off, so installing NTP at the earliest opportunity is a good idea.

The hardware isn’t going to win any prizes for speed, but it’s quite adequate for development. /proc/cpuinfo is attached at the end.

Also I think Imagination have done a good job seeding the community, setting up a mailing list, IRC, and very extensive wiki pages.

Update: Also, no native OCaml compiler … boo 😦

---

$ cat /proc/cpuinfo 
system type		: CI20
processor		: 0
cpu model		: Ingenic Xburst V4.15  FPU V0.0
BogoMIPS		: 1196.85
wait instruction	: yes
microsecond timers	: no
tlb_entries		: 32
extra interrupt vector	: yes
hardware watchpoint	: yes, count: 1, address/irw mask: [0x0fff]
microMIPS		: no
ASEs implemented	: mxu
shadow register sets	: 1
kscratch registers	: 0
core			: 0
VCED exceptions		: not available
VCEI exceptions		: not available
Hardware		: CI20
Serial			: 6f1b2731 1181a003 f9080000 8861aa4f

processor		: 1
cpu model		: Ingenic Xburst V4.15  FPU V0.0
BogoMIPS		: 1196.85
wait instruction	: yes
microsecond timers	: no
tlb_entries		: 32
extra interrupt vector	: yes
hardware watchpoint	: yes, count: 1, address/irw mask: [0x0fff]
microMIPS		: no
ASEs implemented	: mxu
shadow register sets	: 1
kscratch registers	: 0
core			: 0
VCED exceptions		: not available
VCEI exceptions		: not available
Hardware		: CI20
Serial			: 6f1b2731 1181a003 f9080000 8861aa4f

virt-builder Debian 8 (Jessie) image

Debian 8 was released a couple of days ago, and you can now install it through virt-builder.

Use --notes to read the release notes:

$ virt-builder debian-8 --notes

To build an image:

$ virt-builder debian-8 \
    --firstboot-command "dpkg-reconfigure openssh-server"

To boot it under libvirt:

$ virt-install --import \
  --name debian-8 --ram 2048 \
  --disk path=debian-8.img,format=raw --os-variant=debianwheezy

(At some point --os-variant=debianjessie will work, but virt-install doesn’t support it yet)

Update: This is how I ended up running Debian 8:

$ virt-builder debian-8 \
    --size=30G \
    --root-password PASSWORD \
    --edit '/etc/apt/sources.list: s/wheezy/jessie/g' \
    --run-command '
      apt-get -y install debian-keyring debian-archive-keyring
      apt-key update
    ' \
    --install emacs,nfs-common,sudo \
    --edit '/etc/ssh/sshd_config:
              s/^#PermitEmptyPasswords no/PermitEmptyPasswords yes/' \
    --firstboot FIRSTBOOT.sh
    --run-command 'update-rc.d virt-sysprep-firstboot defaults' \
    --run-command 'killall dbus-daemon cgmanager ||:'

New in libguestfs: virt-log

In libguestfs ≥ 1.27.17, there’s a new tool called virt-log for displaying the log files from a disk image or virtual machine:

$ virt-log -a disk.img | less

Previously you could write:

$ virt-cat -a disk.img /var/log/messages

That worked for some Linux guests, but several things happened:

• The binary systemd journal
• The command above doesn’t work on Debian guests (you need to use /var/log/syslog)
• Windows uses the Windows Event Log

Virt-log is designed to do the right thing automatically (although at the moment Windows support is not finished). In particular it will automatically decode and display the systemd journal, and it knows the different locations that some Linux distros store their plain text log files.

Analysis of the size of libguestfs dependencies

In libguestfs ≥ 1.26 we are going to start splitting the package up into smaller dependencies. Since the full libguestfs package has lots of dependencies because it has to be able to process lots of obscure filesystems, the question is how best to split up the dependencies? We could split off, say, XFS support into a subpackage, but how do we know if that will save any space?

Given the set of dependencies, we want to know the incremental cost of adding another dependency.

We can get an exact measure of this by using supermin to build a chroot containing the set of dependencies, and a second chroot containing the set of dependencies + the additional package. Then we simply compare the sizes of the two chroots. The advantage of using supermin is that the exact same script [see end of posting] will work for Fedora and Debian/Ubuntu since supermin hides the complexity of dealing with the different package managers through its package manager abstraction.

The results of this, using the libguestfs appliance dependencies, on Fedora 20, sorted by dependency size, with my comments added:

1. gdisk adds 25420 KB

   This is a surprising result in first place, since gdisk is a fairly small, unassuming C++ program (only ~11KLoC). My initial thought was it must be something to do with being written in C++, but I tested that and it’s not true. The real problem is that gdisk depends on libicu (a Unicode library) which adds 24.6 MB to the appliance. [Note: this issue has been fixed in Rawhide.]

2. lvm2 adds 19432 KB

   The default disk layout of many Linux distros uses LVM so this and similar dependencies have to stay in base libguestfs.

3. binutils adds 16604 KB

   This is a sorry tale. The one file we use from binutils is /usr/bin/strings (33KB). Unfortunately this single binary pulls in a huge dependency (even worse, it’s a development package, and this causes problems on production systems). I don’t really understand why strings is included in binutils.

4. gfs2-utils adds 9648 KB
5. zfs-fuse adds 5208 KB

   Split off in the proposed reorganization.

6. ntfsprogs adds 4572 KB
7. e2fsprogs adds 4312 KB

   Most Linux distros use ext4, and we want to support Windows out of the box, so these are included in base libguestfs.

8. xfsprogs adds 3532 KB

   Split off in the proposed reorganization.

9. iproute adds 3180 KB

   We use /sbin/ip to set up the network card inside the appliance. It’s a shame this “better” replacement for ifconfig is so large.

10. tar adds 2896 KB
11. btrfs-progs adds 2800 KB
12. openssh-clients adds 2428 KB
13. parted adds 2420 KB
14. jfsutils adds 1668 KB
15. genisoimage adds 1644 KB
16. syslinux-extlinux adds 1420 KB
17. augeas-libs adds 1404 KB
18. iputils adds 1128 KB
19. reiserfs-utils adds 1076 KB
20. mdadm adds 1032 KB
21. strace adds 976 KB
22. lsof adds 972 KB
23. vim-minimal adds 912 KB
24. rsync adds 812 KB
25. libldm adds 616 KB
26. psmisc adds 592 KB
27. nilfs-utils adds 520 KB
28. hfsplus-tools adds 480 KB

---

The test script used to produce these results:

#!/bin/bash -

# NB: For this program to work, you must have the following
# packages (or as many as possible) installed locally.
pkgs='acl attr augeas-libs bash binutils bsdmainutils btrfs-progs
bzip2 coreutils cpio cryptsetup cryptsetup-luks diffutils dosfstools
e2fsprogs extlinux file findutils gawk gdisk genisoimage gfs2-utils
grep grub grub-pc gzip hfsplus hfsplus-tools hivex iproute iputils
jfsutils kernel kmod less libaugeas0 libcap libcap2 libhivex0 libldm
libpcre3 libselinux libsystemd-id128-0 libsystemd-journal0 libxml2
libyajl2 linux-image lsof lsscsi lvm2 lzop mdadm module-init-tools
mtools nilfs-utils ntfs-3g ntfsprogs openssh-clients parted pcre
procps procps-ng psmisc reiserfs-utils reiserfsprogs rsync scrub sed
strace syslinux syslinux-extlinux systemd sysvinit tar udev ufsutils
util-linux util-linux-ng vim-minimal vim-tiny xfsprogs xz xz-utils
yajl zerofree zfs-fuse'

# These are the packages (from the above list) that we want to test.
testpkgs="$pkgs"

# Helper function to construct an appliance and see how big it is.
function appliance_size
{
    set -e
    supermin --prepare -o /tmp/supermin.d "$@" >&/dev/null
    supermin --build -f chroot -o /tmp/appliance.d \
      /tmp/supermin.d >&/dev/null
    du -s /tmp/appliance.d | awk '{print $1}'
}

# Construct entire appliance to see how big that would be.
totalsize=`appliance_size $pkgs`

# Remove each package from the list in turn, and find out
# how much extra that package contributes.
for p in $testpkgs; do
    opkgs=
    for o in $pkgs; do
        if [ $o != $p ]; then opkgs="$opkgs $o"; fi
    done
    size=`appliance_size $opkgs`
    extra=$(($totalsize - $size))

    echo $p adds $extra KB
done

Debian/Ubuntu package for Oz (an excellent VM builder)

Oz is an excellent virtual machine builder. I reviewed it here.

This is a temporary Debian (and possibly Ubuntu) package for Oz:

http://oirase.annexia.org/tmp/oz_0.7.0-4_all.deb

I’ll put up a more permanent site for Debian packages later when we’ve decided where to host them.

Note there is no separate source archive of this right now. The source is upstream git + the Debian patch that I posted on the mailing list (but the mailing list archives seem to be broken — fixing it). You can build the exact same package yourself by checking out git, applying the Debian patch, and doing:

debuild -i -uc -us -b

libguestfs 1.12.6 for Debian

Thanks to the tireless work of Hilko Bengen, libguestfs 1.12.6 is now available as an official Debian package.

Also, you can compile hivex on Mac OS X and Windows, thanks to Alex Nelson and Gillen Daniel respectively.

virt-manager + libguestfs in Debian

Guido reports that virt-manager and libguestfs from Debian/unstable are able to properly inspect VMs, displaying icons and listing applications.

Thanks especially to Hilko Bengen, Cole Robinson, Guido Gunther, librarian, a small army of Fedora and Debian testers, and Red Hat for paying me and Cole to write this.

Debian packages for libguestfs 1.12

Debian packages for libguestfs 1.12 are now available. You must read the README file before trying to use these.