32 or 64 bit virtual CPU in KVM

Most hosts these days are 64 bit and have been for a few years. In fact, 32 bit hardware which supports the hardware-assisted virtualization needed for KVM is a rare thing. I have an old laptop which has this rare combination.

However 32 bit guests are relatively common. You might need to run Windows XP, or a proprietary kernel module that only works with a 32 bit kernel, and virtualization is an ideal way to keep these legacy programs going.

So the common question I am asked is: When installing a 32 bit guest, should I choose an i686 or x86-64 (32 or 64 bit) architecture?

The answer seems obvious. You should select i686 for your 32 bit guest.

Actually, no, this isn’t right. x86-64 is the default, and if you try it out you’ll see that 32 bit guests work just fine.

The reason that a 32 bit guest works even though the architecture is x86-64 is simple: When AMD first invented the 64 bit extensions that we now call x86-64, like all previous extensions to the 8086 model, they made them backwards compatible. Like real PC hardware, KVM x86-64 runs 32 bit code (and 16 bit, and 8 bit) code just fine. You can still run MS-DOS on your new x86-64 hardware, even though it’s a 16 bit program. You can still install Windows XP on your x86-64 hardware even though that is a 32 bit program. Same applies to x86-64 KVM guests.

What does the architecture selection do? It changes the -cpu flag that we pass to qemu-kvm. Look at the range of CPUs that qemu-kvm supports:

$ qemu-kvm -cpu \?
x86       Opteron_G3
x86       Opteron_G2
x86       Opteron_G1
x86          Nehalem
x86           Penryn
x86           Conroe
x86           [n270]
x86         [athlon]
x86       [pentium3]
x86       [pentium2]
x86        [pentium]
x86            [486]
x86        [coreduo]
x86          [kvm32]
x86         [qemu32]
x86          [kvm64]
x86       [core2duo]
x86         [phenom]
x86         [qemu64]
x86           [host]

What happens in detail is that for x86-64, no -cpu flag is passed, which is the same as using -cpu qemu64. If the architecture is set to i686, then -cpu qemu32 flag is passed. You can see qemu64 and qemu32 appearing in the list above.

This raises two questions: What does the -cpu flag do, and what are “qemu64″ and “qemu32″ (not a CPU that has rolled off an Intel or AMD assembly line …).

Both questions can be answered by using a “super secret!” QEMU command line flag, -cpu ?dump. Dumping out the two CPU models of interest to us:

$ qemu-kvm -cpu ?dump
[...]
x86         [qemu32]  QEMU Virtual CPU version 0.12.91
  family 6 model 3 stepping 3 level 4 xlevel 0x80000004 vendor ""
  feature_edx 0781abfd (sse2 sse fxsr mmx pat cmov pge sep apic cx8 mce pae msr tsc pse de fpu)
  feature_ecx 00800001 (popcnt pni|sse3)
  extfeature_edx 00000000 ()
  extfeature_ecx 00000000 ()
[...]
x86         [qemu64]  QEMU Virtual CPU version 0.12.91
  family 6 model 2 stepping 3 level 4 xlevel 0x8000000a vendor "AuthenticAMD"
  feature_edx 078bfbfd (sse2 sse fxsr mmx clflush pse36 pat cmov mca pge mtrr sep apic cx8 mce pae msr tsc pse de fpu)
  feature_ecx 00802001 (popcnt cx16 pni|sse3)
  extfeature_edx 2191abfd (lm fxsr mmx nx pat cmov pge syscall apic cx8 mce pae msr tsc pse de fpu)
  extfeature_ecx 00000065 (sse4a abm svm lahf_lm)

(The “lm” flag means Long Mode, which means 64 bit)

The only thing these flags do is change what is reported back to the guest in the CPUID instruction [PDF link]. The actual processor emulation is not changed*.

Could you install a 64 bit guest even if the architecture was configured as “i686″. In theory it could work, but in reality it won’t because Linux checks very early on if the CPUID instruction reports Long Mode (see the verify_cpu function in arch/x86/kernel). Basically Linux or any sane OS would refuse to boot.

Could you use all the features of the CPU if you just ignored CPUID? Yes, but really, don’t do this. Some badly behaved applications in fact do this (Skype for one, but it’s not the only culprit) and this causes all sorts of problems particularly for live migration, since we can’t predict any more if we can safely migrate a virtual machine over to another host that has different features.

* Two complications which make this statement not quite true: QEMU changes some very minor aspects of emulation based on the CPU features, in particular whether or not to enable APIC. Secondly KVM itself can emulate some instructions which are not supported by the hardware. So what your guest sees is a superset of what the hardware can do.

For more information on CPU models, see:

• http://wiki.qemu.org/Features/CPUModels
• http://www.linux-kvm.org/page/Tuning_KVM
• Dan Berrange’s discussion of CPU models and libvirt

Update #1

1. I think that -cpu qemu32 (or selecting “i686″ in the advanced options of virt-manager) is no earthly use to anyone and I defy anyone to name a serious use case for it.
2. -cpu qemu64 is not the same as “all the features supported by my hardware”. If you want that then you have to use -cpu host. This might even be a good idea, but it’s likely to be incompatible with live migration. If you don’t care about live migration then in theory this will give your guests maximum potential performance.

Tip: Get the hostname of a guest

Because different operating systems store the hostname in different places, you have to know in advance what sort of OS your guest is (perhaps using virt-inspector). Perhaps we should add the hostname to virt-inspector.

This works for Fedora guests:

# virt-cat F13x64 /etc/sysconfig/network | \
  grep HOSTNAME= | \
  awk -F= '{print $2}'
f13x64.home.annexia.org

This is for Debian/Ubuntu guests:

# virt-cat Debian5x64 /etc/hostname
debian5x64.home.annexia.org

For Windows guests:

# virt-win-reg Win7x32 \
  'HKEY_LOCAL_MACHINE\System\ControlSet001\Services\Tcpip\Parameters' \
  Hostname
win7x32

I’m not completely clear how to get the DNS domain name from Windows. According to this article you should just replace “Hostname” with “Domain” in the above command, but for me that yields just an empty string.

Linux core_pattern fail

1. core_pattern obeys the current chroot, which effectively makes it useless as a way to collect cores centrally if your system has any chrooted processes.
2. core_pattern explicitly prevents you from dumping to non-regular files. In a virtual environment it’d be very useful to coredump to a block device.

I’m at the KVM Forum and LinuxCon Boston

I will be at the KVM Forum and LinuxCon from 9th August in Boston, USA.

Tip: daily Bugzilla reports

This is my inbox, and it sucks:

Bugzilla is like a black hole for bugs. The search tools fail so badly it’s often impossible to find a bug that you were working on the day before. It’s slow, clumsy, and disorganized.

But one bright point is it has a reasonable command line reporting tool also available as a Fedora package. So I decided yesterday to write a Bugzilla report script and have it email me daily from cron.

The starting point is to identify “bugs of interest”. The bugs of interest to me are:

• bugs I reported
• bugs I am assigned to fix
• bugs I’m CC’d on

I put some thought into this set of criteria for bugs:

1. I should be able to register an interest in any bug: Yes, by adding myself to the CC field.
2. I should be able to unregister an interest: Yes, by removing myself from the CC, or closing or reassigning those bugs where I’m reporter or assignee.
3. It shouldn’t tell me about bugs I’m not interested in: Yes, I would be reporter, assignee or in the CC list for any bug related to me or a package I maintain.
4. It shouldn’t miss out any bugs I might be interested in: Yes, if I’ve ever added a comment, I will be in the CC field at least, and I can always unregister myself from any I no longer care about.

With the command line tool, getting the raw list of bug IDs is very simple.

for flag in -r -c -a; do
  bugzilla query $flag $email -i
done | sort -n -u

(replacing $email with your email address, or even mine if you so care).

That pulls out 781 unique bug IDs, the oldest being one I don’t remember reporting and the most recent being a virt-v2v bug. (Note this includes CLOSED bugs which in the script are ignored).

Now I take the list of bug IDs and pull out the other fields I want from the Bugzilla database. The command below is just an illustrative example of what the script does:

$ bugzilla query -b "86308,601092" \
  --outputformat="%{bug_id}|%{bug_status}|%{product}|%{component}|%{short_desc}" |
  sort -n -t"|"
86308|CLOSED|Red Hat Web Site|Other|Red Hat Command Centre site is down
601092|NEW|Red Hat Enterprise Linux 5|libguestfs|[RFE]Incorrect error msg popped up when missing "-f" in v2v command

As far as I know you have to guess what the %-fields are called. I also had to choose a separator character which wouldn’t appear in any field except the short_desc (which is always the last field), since fields like the product name can and do contain spaces.

The rest of the script is “merely” formatting this whole lot into a nice looking email report:

Bugs in OPEN states:

*** coccinelle ***

  In Fedora:
    502185 coccinelle segfaults on ppc64
    579457 coccinelle-0.2.3rc1 is available

*** e2fsprogs ***

  In Red Hat Enterprise Linux 5:
    518190 mke2fs -t option in RHEL5 differs vs upstream, leading to confusion

[etc]

You can download the script here: http://oirase.annexia.org/tmp/bugs-report.ml.txt

Supercomputing superpowers (BBC article)

BBC article [requires Flash] on the Top500 list. What I found interesting was firstly to what extent Linux dominates, as shown in the diagram below, and also how well the UK did relative to our size and status as a country (mainly due to four huge supercomputers in Edinburgh and at the European Centre for Medium-Range Weather Forecasts in Reading).

The BBC continues to lead the way in educating people about statistics.

Re:boot

How many times do we boot up a Linux kernel when we build and test libguestfs inside koji (the Fedora build system)?

40 times.

libguestfs 1.3.13 is available now for Fedora users …

Quick tip: Learn to love core dumps

Core dumps are one way to track down those segfaults which can’t easily be captured in gdb, either because they are rare and unpredictable, or they happen in some deeply nested subprocess. But no one likes core files being left all over the filesystem. Instead, capture your core dumps in a central directory and analyze them later at your pleasure.

Add this to /etc/rc.local and also run it as a root:

mkdir -p /var/log/core
chmod 0777 /var/log/core
echo "/var/log/core/core.%t.%p.%e" > /proc/sys/kernel/core_pattern
echo 0 > /proc/sys/kernel/core_uses_pid

(By the way, ABRT also uses core_pattern so running these commands will disable ABRT, if you care).

You may need to adjust /etc/security/limits.conf:

*               hard    core            unlimited
*               soft    core            unlimited

On some versions of Fedora, also comment out this line from /etc/profile. (What on earth was the thinking behind this?)

#ulimit -S -c 0 > /dev/null 2>&1

Check that core dumps are enabled. This is what you should see (you may need to log out and log in again):

$ ulimit -Hc
unlimited
$ ulimit -Sc
unlimited

Now, any time a process crashes, it will leave a core file in /var/log/core:

$ ls -l /var/log/core/
total 1800
-rw-------. 1 rjones rjones 303104 May  5 15:25 core.1273069556.28228.ls
[...]
$ gdb /bin/ls /var/log/core/core.1273069556.28228.ls
[...]

This doesn’t enable core dumps for all services, since anything started at boot goes down a different path. If you restart individual services by hand, then core dumps will be enabled for those.

Quick tip: Timing things in a shell script

We all know about the ‘time’ command. That’s great if you have one command that you want to time.

But how about if you want to time several actions together in a shell script, eg:

#!/bin/sh -
start_the_clock
A
B
C
how_long_so_far
D
E
stop_the_clock

The ‘time’ command isn’t so useful for this, unless you want to stick ‘time’ in front of each action A, B, C, D & E and add them up in your head.

But there’s a nice little date/awk trick you can use for this:

#!/bin/sh -
date +%s.%N > /tmp/times
A
B
C
date +%s.%N >> /tmp/times
D
E
date +%s.%N >> /tmp/times
awk '{ if (!start) { start = $1 } else { print $1-start } }' < /tmp/times

which would print something out like this:

25.2957   # the time that actions A-C took
128.529   # the time that actions A-E took

(Thanks Jim Meyering for suggesting date +%s.%N)

RHEL 6 public beta 1

We released the first public beta of Red Hat Enterprise Linux 6 today. You might be able to get it from the official Red Hat FTP site, but it was giving “421 There are too many connected users” errors just now. You can grab this unofficial mirror of the 64 bit DVD ISO instead. The official press release is here.

There are probably a fair few bugs — although not as many as the internal alphas I was playing with last year!

The highlights from my point of view: supported stable libguestfs, libguestfs tools and V2V, OCaml compiler (not supported), virt-top, and hivex.

Edit: Excellent interview with Tim Burke in The Register which also covers the main new features.