QEMU emulator version 6.0.50 Failure with nested FreeBSD bhyve
BUG:
Starting FreeBSD Layer 2 bhyve Guest within Layer 1 FreeBSD VM Host on Layer 0 Ubuntu 21.04 KVM / QEMU Host results in Layer 1 Guest / Host Pausing with "Emulation Failure"
TESTING:
NOTE: This issue was initially raised via Ubuntu Distribution: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1876678 Result of testing with upstream QEMU 6.0.50 was to post as upstream QEMU issue.
Same failure has ocurred in testing with Ubuntu 20.04, 21.04 & 21.04 with upstream QEMU
Historial testing:
- Layer 0 - Ubuntu 20.04 Host
- Layer 1 - FreeBSD 12.1 with OVMF + bhyve hypervisor Guest/Host
- Layer 2 - FreeBSD 12.1 guest
Layer 0 Host is: Ubuntu 20.04 LTS KVM / QEMU / libvirt
QEMU VERSION - Ubuntu 20.04
$ virsh -c qemu:///system version --daemon
Compiled against library: libvirt 6.0.0
Using library: libvirt 6.0.0
Using API: QEMU 6.0.0
Running hypervisor: QEMU 4.2.0
Running against daemon: 6.0.0Testing with Ubuntu 21.04 (Hirsute Hippo).
- Layer 0 - Ubuntu 21.04 Host
- Layer 1 - FreeBSD 12.2 with OVMF + bhyve hypervisor Guest/Host
- Layer 2 - FreeBSD 12.2 guest
Layer 0 Host is: Ubuntu 21.04 KVM / QEMU / libvirt
QEMU VERSION - Ubuntu 21.04
$ sudo virsh version
Compiled against library: libvirt 7.0.0
Using library: libvirt 7.0.0
Using API: QEMU 7.0.0
Running hypervisor: QEMU 5.2.0Current Testing: Ubuntu 21.04 + Upstream QEMU
as per Ubuntu 21.04 but now with:
Upstream QEMU (6.0.50) with following configuration:
$ git clone git://git.qemu.org/qemu.git
$ sudo vi /etc/apt/sources.list
-- edit sources.list to have "# deb-src" lines no more commented out
$ sudo apt update
$ sudo apt build-dep qemu
$ cd qemu
$ mkdir build
$ cd build
$ ../configure --disable-werror --disable-user --disable-linux-user --disable-docs --disable-guest-agent --disable-sdl --disable-gtk --disable-vnc --disable-xen --disable-brlapi --disable-fdt --disable-hax --disable-vde --disable-netmap --disable-rbd --disable-libiscsi --disable-libnfs --disable-smartcard --disable-libusb --disable-usb-redir --disable-seccomp --disable-glusterfs --disable-tpm --disable-numa --disable-opengl --disable-virglrenderer --disable-xfsctl --disable-slirp --disable-blobs --target-list=x86_64-softmmu --disable-rdma --disable-pvrdma --disable-attr --disable-vhost-net --disable-vhost-vsock --disable-vhost-scsi --disable-vhost-crypto --disable-vhost-user --disable-spice --disable-qom-cast-debug --disable-bochs --disable-cloop --disable-dmg --disable-qcow1 --disable-vdi --disable-vvfat --disable-qed --disable-parallels --disable-avx2 --disable-nettle --disable-gnutls --disable-capstone --enable-tools
$ sudo make installQEMU VERSION - Ubuntu 21.04 + Upstream QEMU:
$ qemu-system-x86_64 --version
QEMU emulator version 6.0.50 (v6.0.0-540-g6005ee07c3)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developersNesting Scenario (as above):
- Layer 0 - Ubuntu 21.04 Host
- Layer 1 - FreeBSD 12.2 with OVMF + bhyve hypervisor Guest/Host
- Layer 2 - FreeBSD 12.2 guest
Result when running Layer 2 VM on FreeBSD bhyve Layer 1:
Layer 1 VM goes into pause, when starting Layer 2 quest - as per original test (with Ubuntu 20.04)
LIBVIRT LOG Layer 0 - Ubuntu Host:
2021-05-17 12:31:09.748+0000: starting up libvirt version: 7.0.0, package: 2ubuntu2 (Christian Ehrhardt <christian.ehrhardt@canonical.com> Wed, 07 Apr 2021 13:33:46 +0200), qemu version: 5.2.0Debian 1:5.2+dfsg-9ubuntu3, kernel: 5.11.0-17-generic, hostname: green.in.graphica.com.au
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
HOME=/var/lib/libvirt/qemu/domain-3-hive-dev-freebsd-12. \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-3-hive-dev-freebsd-12./.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-3-hive-dev-freebsd-12./.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-3-hive-dev-freebsd-12./.config \
QEMU_AUDIO_DRV=spice \
/usr/bin/qemu-system-x86_64 \
-name guest=hive-dev-freebsd-12.2,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-3-hive-dev-freebsd-12./master-key.aes \
-blockdev '{"driver":"file","filename":"/usr/share/OVMF/OVMF_CODE_4M.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/home/WHO//OVMF_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-5.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram \
-cpu Broadwell-IBRS,vme=on,ss=on,vmx=on,pdcm=on,f16c=on,rdrand=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,abm=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on \
-m 4096 \
-object memory-backend-ram,id=pc.ram,size=4294967296 \
-overcommit mem-lock=off \
-smp 4,sockets=4,cores=1,threads=1 \
-uuid 459ff0b9-e0d1-44d4-9862-83315419eeee \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=31,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 \
-device pcie-pci-bridge,id=pci.7,bus=pci.1,addr=0x0 \
-device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x1d.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x1d \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x1d.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x1d.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \
-device ide-cd,bus=ide.0,id=sata0-0-0,bootindex=1 \
-blockdev '{"driver":"file","filename":"/home/WHO//VM-HD.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \
-device ide-hd,bus=ide.1,drive=libvirt-1-format,id=sata0-0-1,bootindex=2 \
-netdev tap,fd=33,id=hostnet0 \
-device vmxnet3,netdev=hostnet0,id=net0,mac=52:54:00:c8:8b:95,bus=pci.7,addr=0x1 \
-netdev tap,fd=34,id=hostnet1 \
-device vmxnet3,netdev=hostnet1,id=net1,mac=52:54:00:6c:c9:c1,bus=pci.7,addr=0x2 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev spicevmc,id=charchannel0,name=vdagent \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 \
-device usb-kbd,id=input2,bus=usb.0,port=3 \
-device usb-tablet,id=input3,bus=usb.0,port=4 \
-spice port=5901,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on \
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1 \
-chardev spicevmc,id=charredir0,name=usbredir \
-device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1 \
-chardev spicevmc,id=charredir1,name=usbredir \
-device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=2 \
-device virtio-balloon-pci,id=balloon0,bus=pci.4,addr=0x0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/1 (label charserial0)
KVM internal error. Suberror: 1
emulation failure
RAX=0000000000000000 RBX=0000000000000000 RCX=0000000000000000 RDX=0000000000000f00
RSI=0000000000000000 RDI=0000000000000000 RBP=0000000000000000 RSP=fffffe002dc31700
R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000000 R11=0000000000000000
R12=0000000000000000 R13=0000000000000000 R14=0000000000000000 R15=0000000000000000
RIP=ffffffff828fc5d9 RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =003b 0000000000000000 ffffffff 00c0f300 DPL=3 DS [-WA]
CS =0020 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0028 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]
DS =003b 0000000000000000 ffffffff 00c0f300 DPL=3 DS [-WA]
FS =0013 0000000800b268d0 ffffffff 00c0f300 DPL=3 DS [-WA]
GS =001b ffffffff82611000 ffffffff 00c0f300 DPL=3 DS [-WA]
LDT=0000 0000000000000000 ffffffff 00c00000
TR =0048 ffffffff81f15e08 00002068 00008b00 DPL=0 TSS64-busy
GDT= ffffffff81f1c608 00000067
IDT= ffffffff81f14da0 00000fff
CR0=8005003b CR2=0000000000000000 CR3=00000000517b9152 CR4=003726e0
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
Code=50 4c 8b 67 58 4c 8b 6f 60 4c 8b 77 68 4c 8b 7f 70 48 8b 3f <0f> 01 c2 48 89 e7 b8 02 00 00 00 eb 07 b8 03 00 00 00 eb 00 41 bb 02 00 00 00 74 06 41 bb
2021-05-17T12:33:38.922639Z qemu-system-x86_64: terminating on signal 15 from pid 1871 (/usr/sbin/libvirtd)
2021-05-17 12:33:39.323+0000: shutting down, reason=destroyedDIAGNOSIS:
I have also worked with FreeBSD team on diagnosis and they following issue and work around have been identified.
See here for FreeBSD diagnosis: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=246168
Diagnostics on this bug and it appears to be triggered in nested virtualization case when apic virtualisation is available in Layer 0 HW and then passed forward to Layer 1 VM via Libvirt: .
Testing found that in case where Layer 1 FreeBSD host had this feature, see "VID,PostIntr" in "VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr" from CPU Feature below:
<<START LAYER 1 - FreeBSD CPU Report from dmesg.boot>>
...
...
CPU: Intel Core Processor (Broadwell, IBRS) (2600.09-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x306d2 Family=0x6 Model=0x3d Stepping=2
Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS>
Features2=0xfffa3223<SSE3,PCLMULQDQ,VMX,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended Features=0x1c0fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP>
Structured Extended Features2=0x4<UMIP>
Structured Extended Features3=0xac000400<MD_CLEAR,IBPB,STIBP,ARCH_CAP,SSBD>
XSAVE Features=0x1<XSAVEOPT>
IA32_ARCH_CAPS=0x8<SKIP_L1DFL_VME>
AMD Extended Feature Extensions ID EBX=0x1001000
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
Hypervisor: Origin = "KVMKVMKVM"
...
...<END LAYER 1 - dimes.log>>
In my case with Intel Broadwell chipset this is available, in case of desktop "core i5-8250U" chip- this reports as:
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
For this case HW case, nested: Layer 0 - Ubuntu 20.04, Layer 1 - FreeBSD 12.1 with bhyve, Layer 2 - FreeBSD 12.1 Works.
Workaround is to disable APIC virtual interrupt delivery:
-
Add entry into Layer 1 - FreeBSD Guest / Host: /boot/loader.conf:
hw.vmm.vmx.use_apic_vid=0 -
Reboot
-
Check via sysctl that virtual_interupt_delivery is disabled:
# sysctl hw.vmm.vmx.cap.virtual_interrupt_delivery
hw.vmm.vmx.cap.virtual_interrupt_delivery: 0 <- should be zeroSo issue is triggered by Nested Virtualisation and APIC interupt handling.
Summary:
Based on tesing with Ubuntu team, advise was to post this issue to QEMU upstream team. Apologies for long submisssion, I have tried to provide summary and details of multiple layers and testing across each of Ubuntu and FreeBSD components.
Thank you.
John Hartley.