Enabling Nested Virtualization in libvirt Environments
A good chunk of my day to day programming activities involve running the code I am developing in virtual machines. This led to a problem when I virtualized my desktop:
Curious: Virtualbox only allows 32 bit operating systems to be emulated.
One virtual machine over in my virtualized FreeBSD workstation, I received the following error when attempting to load the
# kldload vmm vmx_init: processor does not support VMX operation module_register_init: MOD_LOAD (vmm, 0xffffffff81e17169, 0) error 6
Curiouser and curiouser. The bhyve wiki suggests this is related to virtualization platform not supporting EPT, i.e. nested virtualization.
This is the kind of problem I hate: here I am, trying to get on with my day and write some code, and now I have to figure out how to enable nested virtualization on
Half A Solution
For once, the documentation for this semi-obscure bit of Linux functionality isn't too terrible. First, make sure that nested virtualization is actually not allowed on the hypervisor:
# cat /sys/module/kvm_intel/parameters/nested N
Having confirmed this, set
kvm-intel.nested=1 for the kernel's boot arguments in
Don't forget to update grub:
After a system reboot, nested virtualization should be allowed:
# cat /sys/module/kvm_intel/parameters/nested Y
Next, we tighten up our guest defintions. KVM offers degrees of fidelity when emulating CPUs, and unless the CPU is perfectly emulated, nested virtualization won't work. They way to fix this is to edit the virtual machine:
# virsh edit net.anserinae.sevenfoldgates
The CPU model is changed from:
After another system reboot, 64-bit virtualization is now possible in Windows:
Unfortunately, when the same changes are applied to my FreeBSD VM a new, equally inscrutable error is seen when attempting to active
$ kldload vmm vmx_init: processor does not support desired basic capabilities module_register_init: MOD_LOAD (vmm, 0xffffffff81a18592, 0) error 22
According to this PR, the error occurs because
bhyve is a little too zealous in verifying CPU fidelity. This comment caught my eye though:
- Ubuntu 17.04 (kernel 4.10) works without any changes to bhyve
- Ubuntu 15.10 doesn't advertize INVVPID capability. While it would be possible to modify bhyve to work in this mode, I'm loathe to do it since it's such a niche case
- Ubuntu 16.04 seems to have the capabilities that most folk have reported.
The quality of
host-passthrough implementation seems to be improving over time; could upgrading my hypervisor kernel to a sufficiently recent version do the trick? I followed this guide to upgrade my Debian Stretch installation from kernel version 4.9 to 4.11, and saw the following upon attempting to reload the
vmm module in FreeBSD after a hypervisor reboot:
# kldload vmm #
No news is good news, and I was subsequently able to run another virtual machine inside the FreeBSD VM.
bhyve's relatively short development history means that odd bugs are likely to be encountered in more exotic virtualization situations. In this case, good problem reporting and documentation meant that I was able to find a workaround without wasting too much time on upgrade hell.