Enabling Nested Virtualization in libvirt Environments

Friction

A good chunk of my day to day programming activities involve running the code I am developing in virtual machines. This led to a problem when I virtualized my desktop:

Only 32 Bits

Curious: Virtualbox only allows 32 bit operating systems to be emulated.

One virtual machine over in my virtualized FreeBSD workstation, I received the following error when attempting to load the bhyve hypervisor:

# kldload vmm
vmx_init: processor does not support VMX operation
module_register_init: MOD_LOAD (vmm, 0xffffffff81e17169, 0) error 6

Curiouser and curiouser. The bhyve wiki suggests this is related to virtualization platform not supporting EPT, i.e. nested virtualization.

This is the kind of problem I hate: here I am, trying to get on with my day and write some code, and now I have to figure out how to enable nested virtualization on libvirt.

Half A Solution

For once, the documentation for this semi-obscure bit of Linux functionality isn't too terrible. First, make sure that nested virtualization is actually not allowed on the hypervisor:

# cat /sys/module/kvm_intel/parameters/nested
N

Having confirmed this, set kvm-intel.nested=1 for the kernel's boot arguments in /etc/default/grub:

GRUB_CMDLINE_LINUX="reboot=acpi kvm-intel.nested=1"

Don't forget to update grub:

# update-grub

After a system reboot, nested virtualization should be allowed:

# cat /sys/module/kvm_intel/parameters/nested
Y

Next, we tighten up our guest defintions. KVM offers degrees of fidelity when emulating CPUs, and unless the CPU is perfectly emulated, nested virtualization won't work. They way to fix this is to edit the virtual machine:

# virsh edit net.anserinae.sevenfoldgates

The CPU model is changed from:

<cpu mode='host-model'/>

...to:

<cpu mode='host-passthrough'/>

After another system reboot, 64-bit virtualization is now possible in Windows:

64 bits yay

FreeBSD

Unfortunately, when the same changes are applied to my FreeBSD VM a new, equally inscrutable error is seen when attempting to active bhyve:

$ kldload vmm
vmx_init: processor does not support desired basic capabilities
module_register_init: MOD_LOAD (vmm, 0xffffffff81a18592, 0) error 22

According to this PR, the error occurs because bhyve is a little too zealous in verifying CPU fidelity. This comment caught my eye though:

Ubuntu 17.04 (kernel 4.10) works without any changes to bhyve

Ubuntu 15.10 doesn't advertize INVVPID capability. While it would be possible to modify bhyve to work in this mode, I'm loathe to do it since it's such a niche case

Ubuntu 16.04 seems to have the capabilities that most folk have reported.

The quality of libvirt's host-passthrough implementation seems to be improving over time; could upgrading my hypervisor kernel to a sufficiently recent version do the trick? I followed this guide to upgrade my Debian Stretch installation from kernel version 4.9 to 4.11, and saw the following upon attempting to reload the vmm module in FreeBSD after a hypervisor reboot:

# kldload vmm
#

No news is good news, and I was subsequently able to run another virtual machine inside the FreeBSD VM.

Conclusion

bhyve's relatively short development history means that odd bugs are likely to be encountered in more exotic virtualization situations. In this case, good problem reporting and documentation meant that I was able to find a workaround without wasting too much time on upgrade hell.