> **This is not a new bug report, and it is not a request for support on a distro > package.** The bug is already fixed upstream by commit 4af976ef398e. I have filed with > the distribution (Ubuntu) as the template asks — LP #2166503, linked below. I am > raising this here to ask a question only upstream can answer: whether a stable-8.2 > release carrying this commit is possible, and if not, what guidance you would give > distributions still shipping 8.2 so I can relay it to them. > > The problem does **not** reproduce on qemu.git master, because master contains the fix. ## Host environment - Operating system: Ubuntu 24.04.4 LTS - OS/kernel version: Linux 6.8.0-124-generic #124-Ubuntu SMP PREEMPT_DYNAMIC x86_64 - Architecture: x86_64 - QEMU flavor: qemu-system-x86_64 - QEMU version: 8.2.2 (Debian 1:8.2.2+ds-0ubuntu1.16) Deployment note: this is OpenStack 2024.2 via Kolla-Ansible, so libvirt and QEMU run in an OCI container built `FROM ubuntu:noble`. The packages are the unmodified ones from the Ubuntu noble archive (`qemu-system-x86`, `qemu-block-extra`, `1:8.2.2+ds-0ubuntu1.16`). The host is also Ubuntu 24.04 and the kernel above is shared by host and container. Nothing Kolla does modifies QEMU. - QEMU command line: ``` qemu-system-x86_64 -M q35 -m 4096 -enable-kvm \ -blockdev '{"driver":"rbd","pool":"volumes","image":"testvol", "server":[{"host":"10.0.220.1","port":"6789"}], "user":"cinder","node-name":"disk0"}' \ -device virtio-blk-pci,drive=disk0 \ -qmp unix:/tmp/qmp.sock,server=on,wait=off ``` (In production this is generated by libvirt under OpenStack/Kolla; the `-blockdev` line above is the relevant part.) ## Emulated/Virtualized environment - Operating system: Ubuntu 24.04 guest - OS/kernel version: stock Ubuntu 24.04 cloud image kernel - Architecture: x86_64 ## Description of problem `qemu_rbd_get_specific_info()` (`block/rbd.c:1411`) issues a **synchronous, blocking** `rbd_read()` of 8 bytes to probe the image's encryption format. It runs on the main thread with the BQL held — `bdrv_named_nodes_list()` (`block.c:6340`) begins with `GLOBAL_STATE_CODE()` and calls `bdrv_block_device_info()` for every node, which calls `bdrv_get_specific_info()` (`block/qapi.c:265`). If that read is slow — queued behind other I/O, throttled by librbd QoS, or on a degraded cluster — the BQL is held for the whole duration and every vCPU stalls at its next MMIO exit. The guest is completely unresponsive: its clock loses that time and the console reports `rcu_preempt kthread starved`. Two points that may not be obvious from the original commit message, which frames this around the server going away: **1. It is reachable during entirely normal operation.** libvirt sends `query-named-block-nodes` unconditionally alongside `query-blockstats` whenever block stats are requested — `qemuDomainGetStatsBlock()` calls `qemuMonitorGetAllBlockStatsInfo()` and then `qemuMonitorBlockStatsUpdateCapacityBlockdev()` inside the same monitor section, with no flag to skip the second. So `virsh domstats --block`, OpenStack, and any Prometheus libvirt exporter polling a compute node all trigger it, on a perfectly healthy cluster. **2. It is the only network call on that path.** Everything else `bdrv_do_query_node_info()` does for an rbd node is served from memory: `bdrv_getlength()` returns the cached `bs->total_sectors` (rbd never sets `has_variable_length`); `bdrv_get_allocated_file_size()` returns `-ENOTSUP` since rbd is a protocol driver without the hook; `qemu_rbd_co_get_info()` returns the cached `object_size`. Removing this one call leaves no I/O under the BQL at all. The size of the read is irrelevant — it is a synchronous round-trip, so it inherits whatever queue depth the image has. librbd's QoS layer is a strict FIFO, so the 8-byte probe queues behind whatever is already throttled. ## Steps to reproduce 1. Attach an RBD image to a guest as a virtio-blk device. 2. Throttle it so I/O queues: `rbd config image set <pool>/<image> rbd_qos_bps_limit 8388608` Verify enforcement from inside the guest — a configured limit is not always live: `dd if=/dev/vdb of=/dev/null bs=1M count=64 iflag=direct` must report ~8 MB/s. Repeat a few times; do not continue until it is consistent. 3. In the guest, fill the queue: `mkfs.xfs -f /dev/vdb` 4. On the host, while that is still in flight: `time virsh domstats --block <domain>` 5. In the guest, in a second session: `for i in $(seq 60); do date +%H:%M:%S; sleep 1; done` ## Measured results | condition (QEMU 8.2.2) | `virsh domstats --block` | |---|---| | discard in flight | **60.39 s** — guest frozen for the duration | | 256 MiB throttled read in flight | 10.9 s | | same load, `query-blockstats` instead (control) | 0.142 s | The control row is the important one: the same VM under the same load answers the other block-stats command in 0.142 s, so the cost is specific to the command that reaches `bdrv_get_specific_info()` — not to Ceph being slow. Step 5 loses the corresponding seconds from the guest clock. ## Verification that 4af976ef398e fixes it I backported the commit onto Ubuntu's 8.2.2 source. **It applies with no conflicts** — 9 hunks in `block/rbd.c` and 1 in `qapi/block-core.json`, line offsets only. I rebuilt `qemu-block-extra` and replaced only `block-rbd.so`, changing nothing else: same host, same kernel, same Ceph cluster, same volume, same QoS limit, same `mkfs` workload. | test (patched) | result | |---|---| | `virsh domstats --block`, discard in flight | 0.151 / 0.150 / 0.141 / 0.153 / 0.150 / 0.144 s | | exporter scrape, discard in flight | 30 consecutive samples, all 0.19–0.24 s | | guest clock, 60 s under load | continuous, no gaps | So: **60.39 s → ~0.15 s**, with that commit as the only variable. One note for anyone reproducing this: the rebuilt package version must be left exactly `1:8.2.2+ds-0ubuntu1.16`. `util/module.c` requires the module to export `qemu_stamp<CONFIG_STAMP>`, and `scripts/qemu-stamp.py` derives CONFIG_STAMP from a SHA-1 over the meson version, the pkgversion, and the contents of `configure`. Bumping the version with `dch --local` changes the stamp and the module is rejected with "Only modules from the same build can be loaded." followed by "Unknown driver 'rbd'". ## Additional information **Why I am asking about 8.2 at all.** Ubuntu 24.04 LTS ships `1:8.2.2+ds-0ubuntu1.16` and is supported until 2029. The fix is in 10.0.4 and 10.1.0, but `stable-8.2` ended at v8.2.10 (2025-03-26) — 138 days before this commit was authored — so no 8.2 release can carry it. Ubuntu's noble pockets (release/updates/proposed/backports) and the Ubuntu Cloud Archive all ship 8.2.2 with no rbd backport. **Already filed with the distribution**, as the reporting guidelines direct: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/2166503 **Red Hat has backported the same commit** for CentOS Stream 10 / RHEL (RHEL-105440): https://gitlab.com/redhat/centos-stream/src/qemu-kvm/-/merge_requests/399 **My questions:** 1. Is a stable-8.2 release containing this commit possible? 2. If not, is there guidance for distributions in this position that I can relay to Ubuntu? A clear statement that 8.2 is closed and this must be handled downstream would itself be useful to us.