Nested KVM on the Power9 processor works, and it is supported in recent Linux kernels and QEMU versions, allowing you to run a KVM guest that itself runs KVM and hosts further virtual machines. This article explains what nested virtualization means on POWER9 hardware, how to enable it, what performance to expect, and the common pitfalls you may encounter.
Background: POWER9 Virtualization Basics
POWER9 is IBM’s server processor line (also used in some Talos and Blackbird workstations) that supports hardware-accelerated virtualization through the PowerVM hypervisor and, for Linux, through KVM on the Power Architecture (KVM-HV). In normal operation, KVM on POWER9 runs guest operating systems directly on the CPU using hardware virtualization extensions, with the host Linux kernel acting as the hypervisor.
Nested virtualization adds another layer: inside a KVM guest, you run another KVM hypervisor, which itself creates guests. This is useful for development, testing, and CI environments where you need to emulate a full virtualization stack, or for running container orchestration that itself uses KVM nodes.
How Nested KVM Works on Power9
On x86, nested virtualization requires exposing virtualization features to the guest CPU. On POWER9, the situation is different because the Power Architecture uses a partition table and hypervisor calls (hcall) rather than VMX/SVM instructions. KVM on POWER9 already runs guests in a way that is closer to a firmware hypervisor, and nested support is implemented by passing through the necessary hypervisor facilities to the guest.
Specifically, since Linux kernel 4.15 (and improved in later versions), KVM on POWER9 supports nested guests by using the kvm_hv module with the nested capability. When enabled, the L1 hypervisor (the first KVM guest) can use the kvm-pr (powerpc) or kvm-hv mode to run its own guests, with the L0 hypervisor (the host) trapping and emulating privileged operations as needed.
In practice, you need to ensure your host kernel and QEMU are recent enough. For example, on a Power9 system running Linux 5.x with QEMU 6.x or later, you can enable nested by adding kvm_hv.nested=1 to the host kernel command line or by setting the module parameter before loading. Then, when creating a guest, you must expose the pseries CPU with the hv flag and the nested capability.
Enabling Nested KVM on Your Power9 Host
To get nested KVM working, follow these general steps:
- Ensure your host runs a Linux kernel that supports nested KVM on Power9 (at least 4.15, but 5.x is recommended).
- Load the
kvm_hvmodule with the nested option:modprobe kvm_hv nested=1(or addoptions kvm_hv nested=1to/etc/modprobe.d/). - Verify the capability is available:
cat /sys/module/kvm_hv/parameters/nestedshould showY. - Create your L1 guest with a CPU model that includes the
hvflag. In QEMU command line, use-cpu hostor-cpu POWER9withhvandnestedproperties. For libvirt, you may need to edit the domain XML to add - Inside the L1 guest, install KVM (e.g.,
qemu-kvmandlibvirt) and ensure thekvm_hvmodule is loaded there as well.
Note that the exact steps depend on your distribution and kernel version. Some older kernels have bugs that prevent nested from working reliably, so testing with a recent kernel is advisable.
Performance and Limitations
Nested virtualization on POWER9 has a performance overhead, as expected. The L0 hypervisor must emulate some hypervisor calls for the L1 guest, and the L1 guest’s KVM has to trap into L0 for certain operations. In practice, the overhead for CPU-bound workloads can be around 10-30% compared to non-nested, but it varies with the workload and the kernel version.
One significant limitation is that not all POWER9 features are available to nested guests. For example, the L1 guest may not be able to use the kvm-hv mode for its own guests if the host does not support the necessary capabilities. In many cases, the L1 guest runs its own KVM in kvm-pr (powerpc) mode, which is slower than kvm-hv because it uses a more emulation-heavy approach. This can result in higher overhead for the L2 guests.
Another limitation is memory and CPU allocation: you must allocate enough resources to the L1 guest to run its own guests, so plan your memory and vCPU counts carefully.
Common Problems and Troubleshooting
When nested KVM on Power9 does not work, it is usually due to one of the following:
- Kernel too old: Nested support was added in 4.15, but many bugs were fixed later. Use a kernel from 5.4 or newer if possible.
- Module parameter not set: The
kvm_hvmodule must be loaded withnested=1. Check/sys/module/kvm_hv/parameters/nested. - Guest CPU model missing the
hvflag: Without this flag, the L1 guest cannot use KVM hardware acceleration. Ensure the guest CPU is configured withhv. - Libvirt XML issues: If using libvirt, the default CPU model may not include the
hvfeature. You may need to setcpu mode='host-passthrough'or explicitly add the feature. - Permissions: The user running QEMU must have access to
/dev/kvminside the L1 guest, and the L1 guest must have KVM enabled in its kernel.
If you encounter errors like “KVM not supported� inside the L1 guest, check the kernel logs (dmesg) on both the host and the guest to see if the nested capability was detected.
Comparing Nested KVM on Power9 vs x86
On x86, nested virtualization is mature and widely used, with support in both Intel and AMD processors. The overhead is relatively low, and most hypervisors (KVM, VMware, Hyper-V) support it well. On Power9, nested support is newer and less polished, but it is functional for development and testing.
One key difference is that on x86, the CPU exposes VMX/SVM instructions that the L1 hypervisor can use directly, with the L0 hypervisor trapping and emulating them. On POWER9, the hypervisor interface is different, and the L1 hypervisor may need to use a different mode (pr vs hv) which affects performance. As of recent kernels, KVM on POWER9 can run nested guests in kvm-hv mode, but this requires the host to support the nested capability and the guest to be configured correctly.
Frequently Asked Questions
Does nested KVM work on all Power9 systems?
Nested KVM requires a recent Linux kernel (4.15+) and a Power9 CPU that supports the necessary hypervisor features. Most Power9 server systems and workstations should work, but it is always best to test on your specific hardware.
What is the performance penalty of nested KVM on Power9?
The overhead can be significant, often 10-30% for CPU-bound workloads, and even higher for I/O-intensive tasks. The exact penalty depends on the kernel version, the workload, and whether the L1 hypervisor uses kvm-hv or kvm-pr mode.
Can I run Docker inside a nested KVM guest?
Yes, you can run Docker inside a nested KVM guest, but you need to ensure that the guest has enough resources and that the kernel supports the required features. Docker itself uses namespaces and cgroups, not KVM, so it will work as long as the guest OS is functional.
Is nested KVM on Power9 suitable for production?
Generally, nested virtualization is not recommended for production due to performance overhead and complexity. It is more suitable for development, testing, and educational purposes. For production, use PowerVM or KVM directly on the host.
How do I check if my kernel supports nested KVM on Power9?
You can check the kernel configuration by looking for CONFIG_KVM and CONFIG_KVM_PPC in /boot/config-$(uname -r). Additionally, after loading the kvm_hv module, check /sys/module/kvm_hv/parameters/nested to see if nested is enabled.