Virtio balloon

After someone asked me a question about “balloons” (in the virtualization sense) today, I noticed that there is not very much documentation around. This post explains what the KVM virtio_balloon driver is all about.

First of all, what is a balloon driver if you’ve never even heard of the concept? It’s a way to give or take RAM from a guest. (In theory at least), if your guest needs more RAM, you can use the balloon driver to give it more RAM. Or if the host needs to take RAM away from guests, it can do so. All of this is done without needing to pause or reboot the guest.

You might think that this would work as a RAM “hot add” feature, rather like hot adding disks to a guest. Although RAM hot add would (IMHO) be much better, currently this is not how ballooning works.

What we have is a kernel driver inside the guest called virtio_balloon. This driver acts like a kind of weird process, either expanding its own memory usage or shrinking down to nearly nothing, as in the diagrams below:

When the balloon driver expands, normal applications running in the guest suddenly have a lot less memory and the guest does the usual things it does when there’s not much memory, including swapping stuff out and starting up the OOM killer. (The balloon itself is non-swappable and un-killable in case you were wondering).

So what’s the point of a kernel driver which wastes memory? There are two points: Firstly, the driver communicates with the host (over the virtio channel), and the host gives it instructions (“expand to this size”, “shrink down now”). The guest cooperates, but doesn’t directly control the balloon.

Secondly, memory pages in the balloon are unmapped from the guest and handed back to the host, so the host can hand them out to other guests. It’s like the guest’s memory has a chunk missing from it:

Libvirt has two settings you can control called currentMemory and maxMemory (“memory” in the libvirt XML):

maxMemory (or just <memory>) is the memory allocated at boot time to a guest. KVM and Xen guests currently cannot exceed this. currentMemory controls what memory you’re requesting to give to the guest’s applications. The balloon fills the rest of the memory and gives it back to the host for the host to use elsewhere.

You can adjust this manually for your guests, either by editing the XML, or by using the virsh setmem command.