## Host environment - Operating system: Ubuntu 22.04.5 LTS - OS/kernel version: Linux 5.15.0-185-generic - Architecture: x86_64 - QEMU flavor: qemu-system-x86_64 - QEMU version: 11.1.50 - QEMU command line: Multi-process QEMU: two processes connected by `socketpair(AF_UNIX, SOCK_STREAM)`. Remote process: ``` qemu-system-x86_64 -machine x-remote -nodefaults \ -device lsi53c895a,id=lsi1 \ -object x-remote-object,id=robj1,devid=lsi1,fd=<remote end of the socketpair> ``` Client process: ``` qemu-system-x86_64 -machine q35,accel=kvm,kernel-irqchip=split -cpu host -m 2048 \ -object memory-backend-memfd,id=sysmem-file,size=2G -numa node,memdev=sysmem-file \ -kernel bzImage -initrd initramfs.cpio.gz \ -append "console=ttyS0 rdinit=/init panic=-1" \ -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off \ -device pcie-root-port,id=rp0,slot=0,chassis=0,bus=pcie.0 \ -device x-pci-proxy-dev,id=lsi1,fd=<proxy end of the socketpair>,bus=rp0 \ -display none -serial file:guest.log \ -qmp unix:qmp.sock,server=on,wait=off -no-reboot ``` ## Emulated/Virtualized environment - Operating system: Linux with a minimal BusyBox initramfs (no distribution userspace) - OS/kernel version: Linux 6.12.0, built from upstream source with CONFIG_HOTPLUG_PCI_PCIE=y and CONFIG_E1000=y - Architecture: x86_64 ## Description of problem When an `x-pci-proxy-dev` is hot-unplugged, `pci_proxy_dev_exit()` does not undo the irqfd registration made by `setup_irqfd()`. With `kernel-irqchip=split`, this leaves a pointer to the `resample` EventNotifier embedded in the `PCIProxyDev` in the global `kvm_resample_fd_list` after the device is freed. The next EOI for that GSI calls `event_notifier_set()` on freed memory, and an ASan build aborts with a heap-use-after-free (excerpt below, full report under "Additional information"). In the reproducer the unplug is requested over QMP (`device_del`) and completed by the guest's pciehp driver. The EOI comes from ordinary interrupt traffic of another device on the same GSI. I am filing this as confidential because I am not sure whether multi-process QEMU (`x-remote`, `x-pci-proxy-dev`) is inside the security boundary. Please reclassify it as needed. Disclosure: I found this problem with an in-house static analysis tool that is part of ongoing, not yet published research on resource-lifecycle bugs in virtual machine monitors. Then, I reviewed the finding manually and with an LLM (Anthropic Claude). The LLM also helped build the reproducer and draft this report. I ran the reproducer myself on the commit listed above, and the results below come from those runs. Code path (master 8967b2fe1b2a): 1. Realize: `setup_irqfd()` (hw/remote/proxy.c:50) calls `proxy_intx_update()` (:29), which calls `kvm_irqchip_add_irqfd_notifier_gsi(kvm_state, &dev->intr, &dev->resample, dev->virq)` (:45). With a split irqchip, `kvm_irqchip_assign_irqfd()` adds `{gsi, &dev->resample}` to `kvm_resample_fd_list` (accel/kvm/kvm-all.c:2495). `setup_irqfd()` also registers `proxy_intx_update()` as the INTx routing notifier (:85). 2. Unplug: `pci_proxy_dev_exit()` (:143-157) closes the channel and calls `event_notifier_cleanup()` on `intr` and `resample`. It does not call `kvm_irqchip_remove_irqfd_notifier_gsi()`, whose deassign path removes the list entry (kvm-all.c:2502), and it does not clear the routing notifier. The `PCIProxyDev` is then freed from the RCU thread (`bus_free_bus_child()`). 3. EOI: `ioapic_eoi_broadcast()` calls `kvm_resample_fd_notify(n)` for every level-triggered pin with the EOI'd vector (hw/intc/ioapic.c:267). `kvm_resample_fd_notify()` finds the stale entry and calls `event_notifier_set()` on it (kvm-all.c:195). By comparison, `proxy_intx_update()` itself removes the irqfd before it re-routes (:35-38), and `vfio_exitfn()` clears its routing notifier on exit (hw/vfio/pci.c:3748). Error seen (ASan build of master; the full report with the free and allocation stacks is under "Additional information"): ``` ==3660994==ERROR: AddressSanitizer: heap-use-after-free on address 0x52200002dd08 at pc 0x5555570ede52 bp 0x7fff72a6b400 sp 0x7fff72a6b3f0 READ of size 1 at 0x52200002dd08 thread T3 #0 0x5555570ede51 in event_notifier_set ../util/event_notifier-posix.c:112 #1 0x555556ca95a8 in kvm_resample_fd_notify ../accel/kvm/kvm-all.c:195 #2 0x555556cccf7f in ioapic_eoi_broadcast ../hw/intc/ioapic.c:267 SUMMARY: AddressSanitizer: heap-use-after-free ../util/event_notifier-posix.c:112 in event_notifier_set ``` Without ASan: on a build of the same commit without `--enable-asan`, the same steps did not crash (3 runs). The `kvm_resample_fd_notify` trace event, which is emitted only when a list entry matches, fired 16, 17 and 16 times for GSI 23 after the unplug, so the stale entry was used on every EOI. Nothing visible happens because `event_notifier_cleanup()` sets `initialized = false` before the object is freed (util/event_notifier-posix.c:94), and `event_notifier_set()` returns early when it reads that value back from the freed memory (:112). If the memory has been reused by then, `event_notifier_set()` reads whatever the new owner stored there; I did not try to reproduce that case. Configuration: I only tested `kernel-irqchip=split`, which is not the q35 default (hw/i386/pc_q35.c:363). `kvm_resample_fd_list` is only used with a split irqchip, and I did not check what happens with `kernel-irqchip=on`. ## Steps to reproduce 1. Build master with `./configure --enable-asan --target-list=x86_64-softmmu` and `make`. 2. Build a guest kernel with CONFIG_HOTPLUG_PCI_PCIE=y and CONFIG_E1000=y, and install a statically linked busybox on the host. 3. Run the script below: `QEMU=./build/qemu-system-x86_64 BZ=/path/to/bzImage ./repro271.sh`. It: 1. builds a minimal initramfs whose init keeps bringing up any new NIC; 2. starts the remote process and the client with the command lines above; 3. waits 24 s for boot, then sends `device_del lsi1`; the guest ejects the proxy through pciehp and QEMU frees the `PCIProxyDev`; 4. sends `device_add e1000,id=net0,bus=rp0`. The e1000 sits behind the same root port and gets the same GSI (23 here). The guest brings it up, the e1000 raises a level-triggered INTx, and the guest's EOI reaches `kvm_resample_fd_notify(23)`. 4. Expected: the client aborts with the ASan report above and the script prints `=> REPRODUCED (heap-use-after-free)`. This happened in 3 of 3 runs on master 8967b2fe1b2a. An emulated e1000 is used in step 3.4 because it does not register a resample entry itself, so the stale entry is the only one for that GSI. <details> <summary>repro271.sh</summary> ```bash #!/bin/bash # Reproducer: heap-use-after-free in kvm_resample_fd_notify() after an # x-pci-proxy-dev is hot-unplugged (q35, kernel-irqchip=split). # # Needs: /dev/kvm, python3, a statically linked busybox, and a guest bzImage # built with CONFIG_HOTPLUG_PCI_PCIE=y and CONFIG_E1000=y. # Usage: QEMU=./build/qemu-system-x86_64 BZ=/path/to/bzImage ./repro271.sh set -u QEMU=${QEMU:-./build/qemu-system-x86_64} BZ=${BZ:?set BZ to a guest bzImage} BUSYBOX=${BUSYBOX:-$(command -v busybox)} WORK=$(mktemp -d) echo "[*] work dir: $WORK" # Minimal initramfs: keep bringing up any NIC that appears, so that the # hot-added e1000 raises level-triggered INTx interrupts. mkdir -p "$WORK/ir"/{bin,proc,sys,dev} cp "$BUSYBOX" "$WORK/ir/bin/busybox" cat > "$WORK/ir/init" <<'INIT' #!/bin/busybox sh /bin/busybox --install -s /bin mount -t proc proc /proc 2>/dev/null mount -t sysfs sysfs /sys 2>/dev/null mount -t devtmpfs dev /dev 2>/dev/null while true; do for n in /sys/class/net/*; do b=$(basename "$n"); [ "$b" = lo ] || ip link set "$b" up 2>/dev/null done sleep 1 done INIT chmod +x "$WORK/ir/init" ( cd "$WORK/ir" && find . | cpio -o -H newc 2>/dev/null | gzip ) > "$WORK/initramfs.cpio.gz" QEMU="$QEMU" BZ="$BZ" WORK="$WORK" python3 - <<'PY' import json, os, socket, subprocess, time QEMU, BZ, WORK = os.environ["QEMU"], os.environ["BZ"], os.environ["WORK"] ps, rs = socket.socketpair(socket.AF_UNIX, socket.SOCK_STREAM) os.set_inheritable(ps.fileno(), True) os.set_inheritable(rs.fileno(), True) # Remote process: hosts the real device. renv = dict(os.environ, ASAN_OPTIONS="detect_leaks=0:abort_on_error=0") remote = subprocess.Popen( [QEMU, "-machine", "x-remote", "-nodefaults", "-device", "lsi53c895a,id=lsi1", "-object", "x-remote-object,id=robj1,devid=lsi1,fd=%d" % rs.fileno()], pass_fds=[rs.fileno()], env=renv, stdout=open(WORK + "/remote.log", "w"), stderr=subprocess.STDOUT) time.sleep(1) # Client process: q35 with split irqchip, proxy behind a PCIe root port. qmp = WORK + "/qmp.sock" cenv = dict(os.environ, ASAN_OPTIONS="detect_leaks=0:abort_on_error=1:" "log_path=%s/asan" % WORK) client = subprocess.Popen( [QEMU, "-machine", "q35,accel=kvm,kernel-irqchip=split", "-cpu", "host", "-m", "2048", "-object", "memory-backend-memfd,id=sysmem-file,size=2G", "-numa", "node,memdev=sysmem-file", "-kernel", BZ, "-initrd", WORK + "/initramfs.cpio.gz", "-append", "console=ttyS0 rdinit=/init panic=-1", "-global", "ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off", "-device", "pcie-root-port,id=rp0,slot=0,chassis=0,bus=pcie.0", "-device", "x-pci-proxy-dev,id=lsi1,fd=%d,bus=rp0" % ps.fileno(), "-trace", "kvm_resample_fd_notify", "-trace", "ioapic_set_irq", "-trace", "ioapic_eoi_broadcast", "-D", WORK + "/trace.log", "-display", "none", "-serial", "file:%s/guest.log" % WORK, "-qmp", "unix:%s,server=on,wait=off" % qmp, "-no-reboot"], pass_fds=[ps.fileno()], env=cenv, stdout=open(WORK + "/client.log", "w"), stderr=subprocess.STDOUT) def qmp_cmd(cmd): s = socket.socket(socket.AF_UNIX) for _ in range(40): try: s.connect(qmp) break except OSError: time.sleep(0.5) s.settimeout(6) s.recv(65536) s.sendall(b'{"execute":"qmp_capabilities"}\n') time.sleep(0.3) s.recv(65536) s.sendall((json.dumps(cmd) + "\n").encode()) time.sleep(1.2) try: print(" qmp:", s.recv(65536).decode().strip()) except OSError: pass s.close() print("[*] waiting for the guest to boot") time.sleep(24) print("[*] device_del lsi1 (the guest ejects it through pciehp)") qmp_cmd({"execute": "device_del", "arguments": {"id": "lsi1"}}) time.sleep(9) print("[*] device_add e1000 on the same root port (same GSI)") qmp_cmd({"execute": "device_add", "arguments": {"driver": "e1000", "id": "net0", "bus": "rp0"}}) time.sleep(14) print("[*] client exit status:", client.poll(), "(None = still running)") for p in (client, remote): p.kill() PY echo "==== result ====" if grep -q heap-use-after-free "$WORK"/asan.* 2>/dev/null; then grep -h -E "ERROR: AddressSanitizer|SUMMARY" "$WORK"/asan.* echo "=> REPRODUCED (heap-use-after-free)" else echo "=> no ASan report" fi echo "kvm_resample_fd_notify for GSI 23: $(grep -c 'gsi 23' "$WORK/trace.log")" echo "ioapic_eoi_broadcast: $(grep -c 'EOI broadcast' "$WORK/trace.log")" echo "[*] logs in $WORK" ``` 1. </details> ## Additional information - The script also enables the `kvm_resample_fd_notify`, `ioapic_set_irq` and `ioapic_eoi_broadcast` trace events and prints how often they fired. On a build without ASan, a non-zero `kvm_resample_fd_notify` count for GSI 23 after the unplug shows the stale entry. - A proposed patch and its test results are below ("Proposed patch"). - No files are attached, because attachments on confidential issues are not confidential. The script, logs and patch are inline. ### ASan report ASan build of master. Thread T3 is a vCPU thread handling the EOI exit, T1 is the RCU thread, and T0 is the main thread. ``` ==3660994==ERROR: AddressSanitizer: heap-use-after-free on address 0x52200002dd08 at pc 0x5555570ede52 bp 0x7fff72a6b400 sp 0x7fff72a6b3f0 READ of size 1 at 0x52200002dd08 thread T3 #0 0x5555570ede51 in event_notifier_set ../util/event_notifier-posix.c:112 #1 0x555556ca95a8 in kvm_resample_fd_notify ../accel/kvm/kvm-all.c:195 #2 0x555556cccf7f in ioapic_eoi_broadcast ../hw/intc/ioapic.c:267 #3 0x555556ab374c in kvm_arch_handle_exit ../target/i386/kvm/kvm.c:6587 #4 0x555556cb84ca in kvm_cpu_exec ../accel/kvm/kvm-all.c:3628 #5 0x555556cbd4cb in kvm_vcpu_thread_fn ../accel/kvm/kvm-accel-ops.c:54 #6 0x5555570f2d7d in qemu_thread_start ../util/qemu-thread-posix.c:414 0x52200002dd08 is located 3080 bytes inside of 5216-byte region [0x52200002d100,0x52200002e560) freed by thread T1 here: #0 0x7ffff767b537 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:127 #1 0x555556ce9611 in object_finalize ../qom/object.c:682 #2 0x555556ce9611 in object_unref ../qom/object.c:1337 #3 0x555556cd864e in bus_free_bus_child ../hw/core/qdev.c:55 #4 0x55555710e2f4 in call_rcu_thread ../util/rcu.c:324 previously allocated by thread T0 here: #0 0x7ffff767b887 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145 #1 0x7ffff71df788 in g_malloc (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x5e788) #2 0x555556cdaddc in qdev_new ../hw/core/qdev.c:150 #3 0x5555569cd701 in qdev_device_add_from_qdict ../system/qdev-monitor.c:717 #4 0x5555569ce29e in qdev_device_add ../system/qdev-monitor.c:765 SUMMARY: AddressSanitizer: heap-use-after-free ../util/event_notifier-posix.c:112 in event_notifier_set ``` In this build `sizeof(PCIProxyDev)` is 5216 and `resample` is at offset 3072, so offset 3080 is `resample.initialized`, the first field `event_notifier_set()` reads. The freed region is the `PCIProxyDev`, which `device_del` removed and the RCU thread freed. ### Proposed patch The patch makes `pci_proxy_dev_exit()` undo what `setup_irqfd()` set up before the notifiers are cleaned up. It clears the INTx routing notifier and, if an irqfd is registered (`dev->virq != -1`), removes it with `kvm_irqchip_remove_irqfd_notifier_gsi()`. With a split irqchip, that call also removes the `kvm_resample_fd_list` entry. This is the same call `proxy_intx_update()` already makes before it re-routes (hw/remote/proxy.c:35-38). Testing: I applied the patch to efa3b9d5ac (hw/remote/proxy.c is identical on master 8967b2fe1b2a), rebuilt with `--enable-asan`, and ran the reproducer above. With the patch there was no ASan report in 3 of 3 runs, with 16-17 EOIs on GSI 23 per run, and `kvm_resample_fd_notify` never fired for GSI 23. Without the patch, on the same tree, every run (3 of 3) hit the heap-use-after-free at the first EOI. Not addressed by this patch: `kvm_resample_fd_remove()` removes the first list entry whose GSI matches (accel/kvm/kvm-all.c:158-173). If several devices share one GSI, it can remove another device's entry. Diff against efa3b9d5ac (it also applies cleanly to master 8967b2fe1b2a): ```diff diff --git a/hw/remote/proxy.c b/hw/remote/proxy.c index e91566509f..7d5003b31b 100644 --- a/hw/remote/proxy.c +++ b/hw/remote/proxy.c @@ -144,6 +144,13 @@ static void pci_proxy_dev_exit(PCIDevice *pdev) { PCIProxyDev *dev = PCI_PROXY_DEV(pdev); + pci_device_set_intx_routing_notifier(pdev, NULL); + + if (dev->virq != -1) { + kvm_irqchip_remove_irqfd_notifier_gsi(kvm_state, &dev->intr, dev->virq); + dev->virq = -1; + } + if (dev->ioc) { qio_channel_close(dev->ioc, NULL); } ```