Regular readers of this blog will of course be familiar with the joys of virtualization. One of those joys is nested virtualization — running a virtual machine in a virtual machine. Nested KVM is a thing too — that is, emulating the virtualization extensions in the CPU so that the second level guest gets at least some of the acceleration benefits that a normal first level guest would get.
My question is: How deeply can you nest KVM?
This is not so easy to test at the moment, so I’ve created a small project / disk image which when booted on KVM will launch a nested guest, which launches a nested guest, and so on until (usually) the host crashes, or you run out of memory, or your patience is exhausted by the poor performance of nested KVM.
The answer, by the way, is just 3 levels [on AMD hardware], which is rather disappointing. Hopefully this will encourage the developers to take a closer look at the bugs in nested virt.
Git repo: http://git.annexia.org/?p=supernested.git;a=summary
Binary images: http://oirase.annexia.org/supernested/
How does this work?
Building a simple appliance is easy. I’m using supermin to do that.
The problem is how does the appliance run another appliance? How do you put the same appliance inside the appliance? Obviously that’s impossible (right?)
The way it works is inside the Lx hypervisor it runs the L(x+1) qemu on /dev/sda, with a protective overlay stored in memory so we don’t disrupt the Lx hypervisor. Since /dev/sda literally is the appliance disk image, this all kinda works.
Richard WM Jones 
I take it that you are saying that you have got an hypervisor within an hypervisro which is it’s self inside an hypervisor (3 levels deep).
Personally that sounds pritty damn cool
I’m sure that is deeper than under Intel (correct me if wrong)??
Please tell us what cases are you expecting to exhibit unwelcome behaviour when you go this deep . Do you need to do that for any thing particular or is it academic curiosity at this point?
Yes.
We certainly need to test it on different hardware. It will undoubtedly make a difference whether it runs on Intel or AMD. However so far I am only testing it on AMD.
The motivation for this is to make sure that nested KVM works, and I believe one way to do that is to exercise unusual nested KVM configurations and also to demonstrate cool things which will encourage the hardcore developers to take a look.
Pingback: Supernested on the QEMU Advent Calendar | Richard WM Jones