32 or 64 bit virtual CPU in KVM

Most hosts these days are 64 bit and have been for a few years. In fact, 32 bit hardware which supports the hardware-assisted virtualization needed for KVM is a rare thing. I have an old laptop which has this rare combination.

However 32 bit guests are relatively common. You might need to run Windows XP, or a proprietary kernel module that only works with a 32 bit kernel, and virtualization is an ideal way to keep these legacy programs going.

So the common question I am asked is: When installing a 32 bit guest, should I choose an i686 or x86-64 (32 or 64 bit) architecture?

The answer seems obvious. You should select i686 for your 32 bit guest.

Actually, no, this isn’t right. x86-64 is the default, and if you try it out you’ll see that 32 bit guests work just fine.

The reason that a 32 bit guest works even though the architecture is x86-64 is simple: When AMD first invented the 64 bit extensions that we now call x86-64, like all previous extensions to the 8086 model, they made them backwards compatible. Like real PC hardware, KVM x86-64 runs 32 bit code (and 16 bit, and 8 bit) code just fine. You can still run MS-DOS on your new x86-64 hardware, even though it’s a 16 bit program. You can still install Windows XP on your x86-64 hardware even though that is a 32 bit program. Same applies to x86-64 KVM guests.

What does the architecture selection do? It changes the -cpu flag that we pass to qemu-kvm. Look at the range of CPUs that qemu-kvm supports:

$ qemu-kvm -cpu \?
x86       Opteron_G3
x86       Opteron_G2
x86       Opteron_G1
x86          Nehalem
x86           Penryn
x86           Conroe
x86           [n270]
x86         [athlon]
x86       [pentium3]
x86       [pentium2]
x86        [pentium]
x86            [486]
x86        [coreduo]
x86          [kvm32]
x86         [qemu32]
x86          [kvm64]
x86       [core2duo]
x86         [phenom]
x86         [qemu64]
x86           [host]

What happens in detail is that for x86-64, no -cpu flag is passed, which is the same as using -cpu qemu64. If the architecture is set to i686, then -cpu qemu32 flag is passed. You can see qemu64 and qemu32 appearing in the list above.

This raises two questions: What does the -cpu flag do, and what are “qemu64” and “qemu32” (not a CPU that has rolled off an Intel or AMD assembly line …).

Both questions can be answered by using a “super secret!” QEMU command line flag, -cpu ?dump. Dumping out the two CPU models of interest to us:

$ qemu-kvm -cpu ?dump
[...]
x86         [qemu32]  QEMU Virtual CPU version 0.12.91
  family 6 model 3 stepping 3 level 4 xlevel 0x80000004 vendor ""
  feature_edx 0781abfd (sse2 sse fxsr mmx pat cmov pge sep apic cx8 mce pae msr tsc pse de fpu)
  feature_ecx 00800001 (popcnt pni|sse3)
  extfeature_edx 00000000 ()
  extfeature_ecx 00000000 ()
[...]
x86         [qemu64]  QEMU Virtual CPU version 0.12.91
  family 6 model 2 stepping 3 level 4 xlevel 0x8000000a vendor "AuthenticAMD"
  feature_edx 078bfbfd (sse2 sse fxsr mmx clflush pse36 pat cmov mca pge mtrr sep apic cx8 mce pae msr tsc pse de fpu)
  feature_ecx 00802001 (popcnt cx16 pni|sse3)
  extfeature_edx 2191abfd (lm fxsr mmx nx pat cmov pge syscall apic cx8 mce pae msr tsc pse de fpu)
  extfeature_ecx 00000065 (sse4a abm svm lahf_lm)

(The “lm” flag means Long Mode, which means 64 bit)

The only thing these flags do is change what is reported back to the guest in the CPUID instruction [PDF link]. The actual processor emulation is not changed*.

Could you install a 64 bit guest even if the architecture was configured as “i686”. In theory it could work, but in reality it won’t because Linux checks very early on if the CPUID instruction reports Long Mode (see the verify_cpu function in arch/x86/kernel). Basically Linux or any sane OS would refuse to boot.

Could you use all the features of the CPU if you just ignored CPUID? Yes, but really, don’t do this. Some badly behaved applications in fact do this (Skype for one, but it’s not the only culprit) and this causes all sorts of problems particularly for live migration, since we can’t predict any more if we can safely migrate a virtual machine over to another host that has different features.

* Two complications which make this statement not quite true: QEMU changes some very minor aspects of emulation based on the CPU features, in particular whether or not to enable APIC. Secondly KVM itself can emulate some instructions which are not supported by the hardware. So what your guest sees is a superset of what the hardware can do.

For more information on CPU models, see:

• http://wiki.qemu.org/Features/CPUModels
• http://www.linux-kvm.org/page/Tuning_KVM
• Dan Berrange’s discussion of CPU models and libvirt

Update #1

1. I think that -cpu qemu32 (or selecting “i686” in the advanced options of virt-manager) is no earthly use to anyone and I defy anyone to name a serious use case for it.
2. -cpu qemu64 is not the same as “all the features supported by my hardware”. If you want that then you have to use -cpu host. This might even be a good idea, but it’s likely to be incompatible with live migration. If you don’t care about live migration then in theory this will give your guests maximum potential performance.