**Note:** While this was observed with distro-packaged QEMU 8.2.2, both bugs exist in identical form on current QEMU master (`util/module.c` as of 2026-03-26). The bugs are in upstream code (`module_load()` and `module_load_dso()`), not in distribution-specific patches. The distribution packaging merely enables `--enable-module-upgrades`, which activates the affected code path. ## Host environment - Operating system: Ubuntu 24.04 (noble) - OS/kernel version: `Linux 6.14.0-37-generic #37~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC x86_64` - Architecture: x86_64 - QEMU flavor: qemu-system-x86_64 - QEMU version: 8.2.2 (Debian 1:8.2.2+ds-0ubuntu1.13); code verified identical on current master - QEMU command line: ``` /usr/bin/qemu-system-x86_64 \ -name guest=instance-XXXX,debug-threads=on \ -S \ -machine pc-q35-8.2,usb=off,dump-guest-core=off,memory-backend=pc.ram,hpet=off,acpi=on \ -accel kvm \ -cpu EPYC-Genoa \ -m size=33554432k \ -object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":34359738368}' \ -smp 12,sockets=12,dies=1,cores=1,threads=1 \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=531,server=on,wait=off \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=utc,driftfix=slew \ -no-shutdown \ -boot strict=on \ -device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \ -device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \ -device '{"driver":"virtio-scsi-pci","id":"scsi0","bus":"pci.2","addr":"0x0"}' \ -blockdev '{"driver":"file","filename":"/var/lib/nova/instances/_base/XXXX","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap","cache":{"direct":true,"no-flush":false}}' \ -blockdev '{"node-name":"libvirt-2-format","read-only":true,"discard":"unmap","cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \ -blockdev '{"driver":"file","filename":"/var/lib/nova/instances/XXXX/disk","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap","cache":{"direct":true,"no-flush":false}}' \ -blockdev '{"node-name":"libvirt-1-format","read-only":false,"discard":"unmap","cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"}' \ -device '{"driver":"scsi-hd","bus":"scsi0.0","channel":0,"scsi-id":0,"lun":0,"device_id":"drive-scsi0-0-0-0","drive":"libvirt-1-format","id":"scsi0-0-0-0","bootindex":1,"write-cache":"on"}' \ -netdev '{"type":"tap","fd":"287","vhost":true,"vhostfd":"519","id":"hostnet0"}' \ -device '{"driver":"virtio-net-pci","host_mtu":1500,"netdev":"hostnet0","id":"net0","mac":"fa:16:3e:XX:XX:XX","bus":"pci.1","addr":"0x0"}' \ -device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.5","addr":"0x0"}' \ -device '{"driver":"virtio-rng-pci","rng":"objrng0","id":"rng0","bus":"pci.6","addr":"0x0"}' \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on ``` Note: The VM is launched by libvirt (via OpenStack Nova) using local qcow2 storage — no RBD at boot time. The bug is triggered when a Ceph RBD volume is hot-attached later via QMP `blockdev-add` with `"driver":"rbd"`, which requires loading `block-rbd.so` at that point. ## Emulated/Virtualized environment - N/A — the bug is in the host-side QEMU module loader, not guest-specific. ## Description of problem There are two bugs in `util/module.c` that interact when `--enable-module-upgrades` is used and a module is first loaded after the on-disk modules have been replaced by a package upgrade while QEMU is still running. ### Bug A: module_load() aborts directory search on load failure The directory search loop in `module_load()` (lines 282–303 on master) only continues to the next directory when `access()` returns `ENOENT` or `ENOTDIR`. When a file exists but `module_load_dso()` fails (e.g. DSO_STAMP_FUN mismatch), the loop falls through to `goto out` instead of trying the next directory: ```c for (i = 0; i < n_dirs; i++) { char *fname = g_strdup_printf("%s/%s%s", dirs[i], module_name, CONFIG_HOST_DSOSUF); int ret = access(fname, F_OK); if (ret != 0 && (errno == ENOENT || errno == ENOTDIR)) { g_free(fname); continue; // ← only case that tries the next directory } else if (ret != 0) { error_setg_errno(errp, errno, "error trying to access %s", fname); } else if (module_load_dso(fname, export_symbols, errp)) { rv = 1; } g_free(fname); goto out; // ← reached on BOTH access error AND dso load failure } ``` With `--enable-module-upgrades`, the search order is: 1. `$QEMU_MODULE_DIR` (typically unset) 2. `CONFIG_QEMU_MODDIR` (e.g. `/usr/lib/x86_64-linux-gnu/qemu/`) 3. `/var/run/qemu/<version>/` (retained modules for the running build) After a package upgrade, directory 2 contains modules from the new build. `access()` succeeds, `module_load_dso()` detects the build mismatch via the `DSO_STAMP_FUN` symbol check and returns false. The loop then hits `goto out` without ever checking directory 3. The `/var/run/qemu/` fallback therefore only works if the module file is completely absent from the system path — not if it exists but belongs to a different build. This defeats the purpose of `--enable-module-upgrades`. ### Bug B: module_load_dso() does not clean up dso_init_list on failure ```c static bool module_load_dso(const char *fname, bool export_symbols, Error **errp) { // ... assert(QTAILQ_EMPTY(&dso_init_list)); // line 165 g_module = g_module_open(fname, flags); // line 171 // ^^^ .so constructors run here, populating dso_init_list if (!g_module_symbol(g_module, DSO_STAMP_FUN_STR, (gpointer *)&sym)) { // Build mismatch detected g_module_close(g_module); // line 186 return false; // line 187 // ^^^ dso_init_list is NOT drained! } // Success path: dso_init_list is drained here (lines 190–198) QTAILQ_FOREACH_SAFE(e, &dso_init_list, node, next) { QTAILQ_REMOVE(&dso_init_list, e, node); g_free(e); } return true; } ``` When `g_module_open()` loads the `.so`, its GCC constructors (`__attribute__((constructor))` via `register_dso_module_init()`) run and add entries to `dso_init_list`. If the `DSO_STAMP_FUN` check then fails, `g_module_close()` unloads the `.so` but `dso_init_list` is not drained. The entries now contain dangling function pointers. On a subsequent module load attempt, `module_load_dso()` hits the assertion at line 165. A retry is possible because `module_load()` removes the module name from `loaded_modules` on failure (line 308). Crash output: ``` failed to initialize module: /usr/lib/x86_64-linux-gnu/qemu/block-rbd.so Only modules from the same build can be loaded. qemu-system-x86_64: util/module.c:165: module_load_dso: Assertion `QTAILQ_EMPTY(&dso_init_list)' failed. ``` ## Steps to reproduce ### Observed in production (Ubuntu 24.04, OpenStack Nova/libvirt/Ceph) This has been reproduced across multiple minor build upgrades (0ubuntu1.11→12 and 0ubuntu1.12→13): 1. An OpenStack instance is running on a compute node with QEMU `1:8.2.2+ds-0ubuntu1.X`. The instance does not use RBD at boot, so `block-rbd.so` is not loaded initially. 2. QEMU is upgraded on the host to `0ubuntu1.(X+1)` while the instance keeps running. 3. `/run/qemu/Debian_1_8.2.2+ds-0ubuntu1.X/block-rbd.so` exists and is readable. 4. A Cinder/Ceph RBD volume is hot-attached to the instance via libvirt `blockdev-add`. 5. First attempt fails: "Unknown driver 'rbd'". 6. Second attempt: QEMU assertion crash. ### Minimal reproduction (without OpenStack) This requires `--enable-module-upgrades` and two builds of QEMU with different `CONFIG_STAMP` values: 1. Build or install QEMU build A. Start a VM without using a module-backed driver: ``` qemu-system-x86_64 -machine q35,accel=kvm -m 512 -nographic \ -drive file=/tmp/test.qcow2,format=qcow2,if=virtio \ -qmp unix:/tmp/qemu-qmp.sock,server,nowait ``` 2. While the VM is running, replace the on-disk modules with build B (or upgrade the package). Ensure retained modules from build A exist under `/var/run/qemu/<build_A_version>/`. 3. Via QMP, attempt to use a module-backed driver (any module from qemu-block-extra works — rbd, ssh, curl, etc.): ```json {"execute": "qmp_capabilities"} {"execute": "blockdev-add", "arguments": { "driver": "rbd", "node-name": "test0", "pool": "dummy", "image": "dummy", "server": [{"host": "127.0.0.1", "port": "6789"}] }} ``` 4. First attempt: QMP returns `"Unknown driver 'rbd'"`. 5. Second attempt: QEMU aborts with `assert(QTAILQ_EMPTY(&dso_init_list))`. Note: Any module from qemu-block-extra (e.g. block-ssh, block-curl) can be used instead of block-rbd — the bug is in the generic module loader. ## Additional information ### Proposed fix **Bug A — continue search on load failure:** ```diff --- a/util/module.c +++ b/util/module.c @@ -295,10 +295,16 @@ int module_load(const char *prefix, const char *name, Error **errp) g_free(fname); continue; } else if (ret != 0) { /* most common is EACCES here */ error_setg_errno(errp, errno, "error trying to access %s", fname); + g_free(fname); + goto out; } else if (module_load_dso(fname, export_symbols, errp)) { rv = 1; /* module successfully loaded */ + g_free(fname); + goto out; } + /* module_load_dso failed (e.g. build mismatch), try next dir */ + error_free(*errp); + *errp = NULL; g_free(fname); - goto out; + continue; } ``` **Bug B — drain dso_init_list on build mismatch:** ```diff --- a/util/module.c +++ b/util/module.c @@ -183,6 +183,11 @@ static bool module_load_dso(const char *fname, bool export_symbols, error_append_hint(errp, "Only modules from the same build can be loaded.\n"); } + /* Drain dso_init_list: constructors may have added entries */ + QTAILQ_FOREACH_SAFE(e, &dso_init_list, node, next) { + QTAILQ_REMOVE(&dso_init_list, e, node); + g_free(e); + } g_module_close(g_module); return false; } ``` Note: the `dso_init_list` drain must happen before `g_module_close()`, since closing the module unmaps the `.so` and the function pointers stored in the list entries would become dangling. ### Impact Any distribution using `--enable-module-upgrades` is affected. Long-running QEMU instances that predate a package upgrade cannot hot-load any module-backed driver. The first attempt fails (driver not found); the second attempt crashes QEMU. This is the primary use case that `--enable-module-upgrades` was designed to support. ### Related - Ubuntu LP #1847361 (Upgrade of qemu binaries causes running instances to be unable to hot-attach) - Ubuntu LP #1913421 (module retention improvements)