## Host environment - Operating system: Ubuntu 22.04.5 LTS - OS/kernel version: Linux 5.15.0-185-generic - Architecture: x86_64 - QEMU flavor: qemu-storage-daemon (`vduse-blk` export) - QEMU version: 11.1.50, git master 8967b2fe1b2a (master as of 2026-10-20). Two builds: `--enable-asan --target-list=x86_64-softmmu --disable-docs --disable-werror`, and the same without `--enable-asan` - QEMU command line: ``` qemu-storage-daemon \ --blockdev null-co,node-name=disk0,size=67108864,latency-ns=6000000000,read-zeroes=on \ --export vduse-blk,id=exp0,node-name=disk0,name=vblk199 ``` The VDUSE device is then bound to vhost_vdpa (`vdpa dev add name vblk199 mgmtdev vduse`) and driven through `/dev/vhost-vdpa-N` by a small userspace client that is included in the reproducer. ## Emulated/Virtualized environment - Operating system: none (no guest). A userspace vhost-vdpa client running on the host stands in for both the guest's virtio-blk driver and QEMU's vhost-vdpa frontend: it builds the ring and the READ request in its own memory, kicks the queue, and issues `VHOST_VDPA_SET_STATUS(0)` (see "Steps to reproduce"). - OS/kernel version: n/a (no guest; the client runs on the host kernel listed above) - Architecture: n/a (no guest) ## Description of problem When a VDUSE device exported with `--export vduse-blk` is reset while a request is still being processed, qemu-storage-daemon aborts: ``` qemu-storage-daemon: ../subprojects/libvduse/libvduse.c:964: vduse_queue_disable: Assertion `vq->inuse == 0' failed. ``` The reset reaches the daemon as `VDUSE_SET_STATUS` with status 0. libvduse then disables every virtqueue, and `vduse_queue_disable()` asserts that no popped element is outstanding. vduse-blk keeps an element popped for as long as its request coroutine waits in the block layer, and nothing on the reset path waits for that coroutine. So a reset that overlaps an outstanding request kills the daemon and every export it serves. The assertion is active in normal builds (QEMU does not support `NDEBUG`), so this is not specific to debug or ASan builds. Code path (master 8967b2fe1b2a): 1. `vduse_queue_pop()` increments `vq->inuse` (`subprojects/libvduse/libvduse.c:789`), and `vduse_blk_vq_handler()` starts one coroutine per element (`block/export/vduse-blk.c:96-115`). The coroutine yields in `blk_co_preadv()` and gives the element back through `vduse_queue_push()` -\> `vduse_queue_flush()` (`libvduse.c:840`) only after the I/O completes. 2. While the coroutine is waiting, the main loop handles `VDUSE_SET_STATUS(0)`: `vduse_dev_handler()` -\> `vduse_dev_stop_dataplane()` (`libvduse.c:1028-1032`, `:992-1005`) -\> `vduse_queue_disable()` for each queue. 3. `vduse_queue_disable()` calls the export's `disable_queue` callback, which only removes the kick fd handler (`vduse-blk.c:146-157`), and then reaches `assert(vq->inuse == 0)` (`libvduse.c:964`). By contrast, a block-layer drain on the same export does wait for in-flight requests: `vduse_blk_drained_poll()` checks `inflight` (`vduse-blk.c:254-260`). The reset path has no such step, and `VduseOps` has no callback that would let libvduse ask the export to quiesce first. Who sends the reset: whoever drives the device. With `vhost-vdpa-device-pci`, QEMU sends it when the guest resets the device (reboot, driver unbind), and also when the VM is stopped. VDUSE does not offer `VHOST_BACKEND_F_SUSPEND`, so `vhost_vdpa_suspend()` falls back to a device reset (`hw/virtio/vhost-vdpa.c:1342-1363`). I checked this QEMU-side path by reading the code only. This host's kernel (5.15) lacks `VHOST_VDPA_GET_VQS_COUNT`, so `vhost-vdpa-device-pci` does not start here, and the reproducer uses a userspace client instead. I am filing this as confidential because I am not sure how it should be classified. The trigger needs guest kernel privileges, and `docs/system/security.rst` treats such asserts in QEMU as self-inflicted. However, the process that aborts is qemu-storage-daemon, which may serve disks to several VMs at once. Please reclassify it as needed. Results with the reproducer below (master 8967b2fe1b2a): | Build | Runs | Result | |-------|------|--------| | `--enable-asan` | 3 | daemon killed by SIGABRT (exit status 134) at `libvduse.c:964` in 3 of 3 | | without ASan | 3 | same, 3 of 3 | In every run the client's `VHOST_VDPA_SET_STATUS(0)` returned EIO after 30.7 to 32.0 s, which matches the kernel's default VDUSE `msg_timeout` of 30 s because the daemon never replies. The READ never completed (its status byte stayed 0xff). ## Steps to reproduce 1. Build qemu-storage-daemon from the commit above (the `vduse-blk` export is built by default on Linux). 2. On a Linux 5.15 or later host with the `vduse` and `vhost_vdpa` modules and the iproute2 `vdpa` tool, run the script below as root: `QSD=./build/storage-daemon/qemu-storage-daemon ./repro199.sh`. 3. The script exports a null-co disk whose requests take 6 s each, binds the VDUSE device to vhost_vdpa, and builds and runs a small vhost-vdpa client that submits one READ and resets the device 2 s later. It then reports whether the daemon aborted, and cleans up. The result is deterministic. The 6 s latency only makes sure the READ is still outstanding when the reset arrives; any reset that overlaps an outstanding request takes the same path. <details> <summary>repro199.sh</summary> ```bash #!/bin/bash # Reproducer: qemu-storage-daemon aborts in vduse_queue_disable() when a # VDUSE device is reset while a vduse-blk request is still in flight. # # Needs: root, the vduse and vhost_vdpa kernel modules (Linux 5.15+), the # iproute2 "vdpa" tool, gcc, python3, and a qemu-storage-daemon built with # the vduse-blk export. # Usage: QSD=./build/storage-daemon/qemu-storage-daemon ./repro199.sh # # The small vhost-vdpa client below does what a guest virtio-blk driver and # QEMU's vhost-vdpa frontend would do together. It submits one READ, waits # 2 s, then resets the device with VHOST_VDPA_SET_STATUS(0). The null-co backend delays every request by 6 s, # so the READ is still in flight when the reset reaches the daemon. set -u QSD=${QSD:-./build/storage-daemon/qemu-storage-daemon} NAME=vblk199 [ "$(id -u)" = 0 ] || { echo "run as root"; exit 1; } WORK=$(mktemp -d) echo "[*] work dir: $WORK" cat > "$WORK/vdpa_reset.c" <<'EOF' /* Minimal vhost-vdpa virtio-blk client: one READ, then a device reset. */ #include <errno.h> #include <fcntl.h> #include <stdint.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <sys/eventfd.h> #include <sys/ioctl.h> #include <sys/mman.h> #include <time.h> #include <unistd.h> #include <linux/vhost.h> #include <linux/virtio_blk.h> #include <linux/virtio_config.h> #include <linux/virtio_ring.h> static int vfd; static void add_status(uint8_t bit) { uint8_t s = 0; ioctl(vfd, VHOST_VDPA_GET_STATUS, &s); s |= bit; if (ioctl(vfd, VHOST_VDPA_SET_STATUS, &s)) { perror("VHOST_VDPA_SET_STATUS"); exit(1); } } int main(int argc, char **argv) { if (argc != 3) { fprintf(stderr, "usage: %s /dev/vhost-vdpa-N delay_ms\n", argv[0]); return 2; } vfd = open(argv[1], O_RDWR); if (vfd < 0 || ioctl(vfd, VHOST_SET_OWNER, NULL)) { perror(argv[1]); return 1; } /* Echo back the backend features (IOTLB v2); some kernels reject this. */ uint64_t bf = 0; ioctl(vfd, VHOST_GET_BACKEND_FEATURES, &bf); ioctl(vfd, VHOST_SET_BACKEND_FEATURES, &bf); struct vhost_vdpa_iova_range range = { 0 }; ioctl(vfd, VHOST_VDPA_GET_IOVA_RANGE, &range); uint64_t iova = range.first > 0x100000 ? range.first : 0x100000; uint64_t features = 0; ioctl(vfd, VHOST_GET_FEATURES, &features); features &= (1ULL << VIRTIO_F_VERSION_1) | (1ULL << VIRTIO_F_ACCESS_PLATFORM); add_status(VIRTIO_CONFIG_S_ACKNOWLEDGE); add_status(VIRTIO_CONFIG_S_DRIVER); if (ioctl(vfd, VHOST_SET_FEATURES, &features)) { perror("VHOST_SET_FEATURES"); return 1; } add_status(VIRTIO_CONFIG_S_FEATURES_OK); /* One buffer, mapped 1:1 at "iova": rings, then the request. */ size_t len = 0x10000; char *buf = mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0); if (buf == MAP_FAILED) { perror("mmap"); return 1; } struct vhost_msg_v2 msg = { .type = VHOST_IOTLB_MSG_V2 }; msg.iotlb.iova = iova; msg.iotlb.size = len; msg.iotlb.uaddr = (uintptr_t)buf; msg.iotlb.perm = VHOST_ACCESS_RW; msg.iotlb.type = VHOST_IOTLB_UPDATE; if (write(vfd, &msg, sizeof(msg)) != sizeof(msg)) { perror("IOTLB update"); return 1; } struct vring_desc *desc = (void *)buf; struct vring_avail *avail = (void *)(buf + 0x1000); struct virtio_blk_outhdr *hdr = (void *)(buf + 0x3000); uint8_t *status = (uint8_t *)buf + 0x5000; hdr->type = VIRTIO_BLK_T_IN; hdr->sector = 0; *status = 0xff; desc[0] = (struct vring_desc){ iova + 0x3000, sizeof(*hdr), VRING_DESC_F_NEXT, 1 }; desc[1] = (struct vring_desc){ iova + 0x4000, 4096, VRING_DESC_F_NEXT | VRING_DESC_F_WRITE, 2 }; desc[2] = (struct vring_desc){ iova + 0x5000, 1, VRING_DESC_F_WRITE, 0 }; avail->ring[0] = 0; avail->idx = 1; struct vhost_vring_state num = { 0, 16 }, base = { 0, 0 }, enable = { 0, 1 }; struct vhost_vring_addr addr = { .index = 0, .desc_user_addr = iova, .avail_user_addr = iova + 0x1000, .used_user_addr = iova + 0x2000, }; int kick = eventfd(0, 0), call = eventfd(0, 0); struct vhost_vring_file kickf = { 0, kick }, callf = { 0, call }; if (ioctl(vfd, VHOST_SET_VRING_NUM, &num) || ioctl(vfd, VHOST_SET_VRING_BASE, &base) || ioctl(vfd, VHOST_SET_VRING_ADDR, &addr) || ioctl(vfd, VHOST_SET_VRING_KICK, &kickf) || ioctl(vfd, VHOST_SET_VRING_CALL, &callf) || ioctl(vfd, VHOST_VDPA_SET_VRING_ENABLE, &enable)) { perror("vring setup"); return 1; } add_status(VIRTIO_CONFIG_S_DRIVER_OK); uint64_t one = 1; if (write(kick, &one, sizeof(one)) != sizeof(one)) { perror("kick"); return 1; } printf("[drv] READ submitted, resetting the device in %s ms\n", argv[2]); usleep(atoi(argv[2]) * 1000); uint8_t zero = 0; struct timespec t0, t1; clock_gettime(CLOCK_MONOTONIC, &t0); int rc = ioctl(vfd, VHOST_VDPA_SET_STATUS, &zero); int err = errno; clock_gettime(CLOCK_MONOTONIC, &t1); printf("[drv] VHOST_VDPA_SET_STATUS(0) returned %d (%s) after %.1f s\n", rc, rc ? strerror(err) : "ok", (t1.tv_sec - t0.tv_sec) + (t1.tv_nsec - t0.tv_nsec) / 1e9); printf("[drv] request status byte: 0x%02x (0xff = not completed)\n", *status); return 0; } EOF gcc -O1 -Wall -o "$WORK/vdpa_reset" "$WORK/vdpa_reset.c" || exit 1 modprobe vduse && modprobe vhost_vdpa || exit 1 QSD_PID= AUTOPROBE=$(cat /sys/bus/vdpa/drivers_autoprobe) cleanup() { vdpa dev del "$NAME" 2>/dev/null if [ -n "$QSD_PID" ]; then kill "$QSD_PID" 2>/dev/null wait "$QSD_PID" 2>/dev/null fi # A daemon that died leaves /dev/vduse/$NAME behind; remove it. if [ -e "/dev/vduse/$NAME" ]; then python3 - "$NAME" <<'PY' import fcntl, os, sys VDUSE_DESTROY_DEV = 0x41008103 # _IOW(0x81, 0x03, char[256]) fd = os.open("/dev/vduse/control", os.O_RDWR) fcntl.ioctl(fd, VDUSE_DESTROY_DEV, sys.argv[1].encode().ljust(256, b"\0")) PY fi echo "$AUTOPROBE" > /sys/bus/vdpa/drivers_autoprobe } trap cleanup EXIT # Export a null-co disk whose requests each take 6 s. ASAN_OPTIONS=detect_leaks=0 "$QSD" \ --blockdev null-co,node-name=disk0,size=67108864,latency-ns=6000000000,read-zeroes=on \ --export vduse-blk,id=exp0,node-name=disk0,name=$NAME \ > "$WORK/qsd.log" 2>&1 & QSD_PID=$! for _ in $(seq 50); do [ -e "/dev/vduse/$NAME" ] && break; sleep 0.1; done [ -e "/dev/vduse/$NAME" ] || { echo "no /dev/vduse/$NAME"; cat "$WORK/qsd.log"; exit 1; } # Bind the device to vhost_vdpa (not virtio_vdpa) to get /dev/vhost-vdpa-N. echo 0 > /sys/bus/vdpa/drivers_autoprobe vdpa dev add name "$NAME" mgmtdev vduse || exit 1 echo "$NAME" > /sys/bus/vdpa/drivers/vhost_vdpa/bind || exit 1 echo "$AUTOPROBE" > /sys/bus/vdpa/drivers_autoprobe VHOST=$(ls -d /sys/bus/vdpa/devices/$NAME/vhost-vdpa-* 2>/dev/null | head -n1) [ -n "$VHOST" ] || { echo "no vhost-vdpa device for $NAME"; exit 1; } VHOST=/dev/$(basename "$VHOST") echo "[*] $NAME -> $VHOST" "$WORK/vdpa_reset" "$VHOST" 2000 sleep 1 echo "==== result ====" if kill -0 "$QSD_PID" 2>/dev/null; then echo "=> qemu-storage-daemon still running (not reproduced)" else wait "$QSD_PID" echo "qemu-storage-daemon exit status: $? (134 = SIGABRT)" QSD_PID= fi grep -h "Assertion" "$WORK/qsd.log" && echo "=> REPRODUCED (assertion failure)" echo "[*] logs in $WORK" ``` </details> ## Additional information - Script output from one run (ASan build): ``` [*] work dir: /tmp/tmp.nz0icbp0Gu [*] vblk199 -> /dev/vhost-vdpa-1 [drv] READ submitted, resetting the device in 2000 ms [drv] VHOST_VDPA_SET_STATUS(0) returned -1 (Input/output error) after 30.7 s [drv] request status byte: 0xff (0xff = not completed) /home/cc/workspace/vmm-review/repro/qemu/reports/repro199_report.sh: line 203: 594958 Aborted (core dumped) ASAN_OPTIONS=detect_leaks=0 "$QSD" --blockdev null-co,node-name=disk0,size=67108864,latency-ns=6000000000,read-zeroes=on --export vduse-blk,id=exp0,node-name=disk0,name=$NAME > "$WORK/qsd.log" 2>&1 ==== result ==== qemu-storage-daemon exit status: 134 (134 = SIGABRT) qemu-storage-daemon: ../subprojects/libvduse/libvduse.c:964: vduse_queue_disable: Assertion `vq->inuse == 0' failed. => REPRODUCED (assertion failure) [*] logs in /tmp/tmp.nz0icbp0Gu ``` - An earlier run on efa3b9d with a real slow disk instead of null-co latency (dm-delay, 6 s per I/O, `cache.direct=on`) aborted at the same assertion. - Removing the assertion would not fix this. From reading the code: after `vduse_dev_stop_dataplane()` has unmapped all IOVA regions (`libvduse.c:1004`), the resumed coroutine writes the request's status byte through an iovec that points into an unmapped region, and `vduse_queue_push()` silently does nothing because `vring.used` is NULL, so `inuse` stays non-zero. - Related, from reading the code (not run): in `vduse_blk_virtio_process_req()`, the error path for a malformed request (`in_len < 0`, `vduse-blk.c:87-90`) frees the request without pushing the element or decrementing `inflight`. One malformed request therefore leaves `inuse` non-zero for good, and the next reset hits the same assertion even with no I/O in flight. A fix that waits for in-flight requests on reset would also need this path to give the element back. - A possible direction, not a patch: before `vduse_dev_stop_dataplane()` disables the queues, have the export stop taking new requests and wait until `inflight` reaches zero (as drain does), and only then reply to `VDUSE_SET_STATUS`. Rejecting the reset is not an alternative, because the kernel's `vduse_vdpa_reset()` resets its side regardless of the reply.