## Host environment - Operating system: **Ubuntu 24.10** - OS/kernel version: **Linux Arc 6.14.8-061408-generic #202505221337 SMP PREEMPT_DYNAMIC Thu May 22 16:00:37 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux** - Architecture: **x86_64** - QEMU flavor: **qemu-system-x86_64** - QEMU version: **QEMU emulator version 10.0.92 (v10.1.0-rc2-9-ga74434580e)** - QEMU command line (iSCSI server setup follows): ``` ./qemu-system-x86_64 -display none -machine q35,accel=kvm -smp 2 -m 1280 -no-reboot -serial stdio -kernel ./kernel -initrd ./initrd.img -device virtio-scsi-pci,id=scsi -drive file=iscsi://127.0.0.1:10000/root/1,format=raw,if=none,id=scsidrive0,cache=none -device scsi-hd,drive=scsidrive0 -append "console=ttyS0 break" ``` ## Emulated/Virtualized environment - Operating system: **Host version is used** - OS/kernel version: **Host version is used** - Architecture: **x86_64** ## Description of problem A VM, connected to a iSCSI server, hangs after this server abrupt disappearance. ## Steps to reproduce with 100% probability for Ubuntu-based host: 1. Create a stub disk with a single partition and any fs (ext4 used as example): ``` dd if=/dev/zero of=./disk.raw bs=1M count=100 fdisk ./disk.raw <<EOF o n p 1 w EOF mkfs.ext4 -F -E offset=1048576 ./disk.raw ``` 2. Bring up an iSCSI server daemon (`apt install tgt` for Ubuntu-based host) on foreground in the dedicated console: ``` TGT_IPC_SOCKET=/tmp/tgtd.sock tgtd --foreground --iscsi portal=127.0.0.1:10000 ``` 3. Add a LUN 1 with the stub disk to the iSCSI server: ``` TGT_IPC_SOCKET=/tmp/tgtd.sock tgtadm --lld iscsi --op new --mode target --tid 1 --targetname root TGT_IPC_SOCKET=/tmp/tgtd.sock tgtadm --lld iscsi --op new --mode logicalunit --device-type disk --tid 1 --lun 1 --backing-store "$(readlink -f ./disk.raw)" TGT_IPC_SOCKET=/tmp/tgtd.sock tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL ``` 4. Copy running kernel and initrd into the current directory because Ubuntu doesn't allow any users except root to read kernel and initrd: ``` sudo cp /boot/vmlinuz ./kernel && sudo chown ${USER}:${USER} ./kernel sudo cp /boot/initrd.img ./initrd.img && sudo chown ${USER}:${USER} ./initrd.img ``` There is not much need from these two apart of basic boot sequence, running busybox and basic mount commands. So, nearly every modern distro kernel+inird pair should do the job. 5. Start a VM with kernel and initrd from the previous step: ``` qemu-system-x86_64 \ -display none \ -machine q35,accel=kvm \ -smp 2 \ -m 1280 \ -no-reboot \ -serial stdio \ -kernel ./kernel \ -initrd ./initrd.img \ -device virtio-scsi-pci,id=scsi \ -drive file=iscsi://127.0.0.1:10000/root/1,format=raw,if=none,id=scsidrive0,cache=none \ -device scsi-hd,drive=scsidrive0 \ -append "console=ttyS0 break" ``` All the boot commands will be printed into the console due to `serial stdio + console=ttyS0`. The busybox should be available via stdout + stdin as well. 6. In the appeared busybox console mount a single **/dev/sda1** partition which should be the only available iSCSI one: ``` Loading, please wait... Starting systemd-udevd version 256.5-2ubuntu3.3 Begin: Loading essential drivers ... done. Spawning shell within the initramfs BusyBox v1.36.1 (Ubuntu 1:1.36.1-9ubuntu1) built-in shell (ash) Enter 'help' for a list of built-in commands. (initramfs) mount / -oremount,rw (initramfs) mkdir /mnt (initramfs) mount /dev/sda1 /mnt [ 12.874551] EXT4-fs (sda1): recovery complete [ 12.874943] EXT4-fs (sda1): mounted filesystem eea7edb3-0a7e-4715-b29a-51f8308af132 r/w with ordered data mode. Quota mode: none. (initramfs) ls /mnt lost+found (initramfs) ``` 7. Close the iSCSI server from the step 2. It should exit after standard CTRL+C signal 8. Try umounting previously mounted **/mnt** from the busybox shell from the step 6: ``` (initramfs) umount /mnt ``` 9. The call hangs and the VM now is could be powered off only by `SIGKILL` . Attempt to sent `SIGINT` to the qemu process results in warning message `qemu-system-x86_64: terminating on signal 2` and the VM entering in some undefined state. ## Additional information: I was able to gather some GDB traces: * After the iSCSI server close and after attempt to umount a dead connection: ``` Thread 5 (Thread 0x78e0cf9ff6c0 (LWP 341221) "qemu-system-x86"): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000654c460c132b in qemu_futex_wait (f=0x654c47449618 <rcu_call_ready_event>, val=4294967295) at /home/zentarim/src/qemu/include/qemu/futex.h:47 #2 0x0000654c460c155f in qemu_event_wait (ev=0x654c47449618 <rcu_call_ready_event>) at ../util/event.c:162 #3 0x0000654c460cd230 in call_rcu_thread (opaque=0x0) at ../util/rcu.c:278 #4 0x0000654c460bfdac in qemu_thread_start (args=0x654c7647f760) at ../util/qemu-thread-posix.c:393 #5 0x000078e0ddaa1e2e in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447 #6 0x000078e0ddb33a4c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 Thread 4 (Thread 0x78e0ceecb6c0 (LWP 341222) "qemu-system-x86"): #0 __GI___ioctl (fd=14, request=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36 #1 0x0000654c45e85982 in kvm_vcpu_ioctl (cpu=0x654c767adc20, type=44672) at ../accel/kvm/kvm-all.c:3403 #2 0x0000654c45e8502b in kvm_cpu_exec (cpu=0x654c767adc20) at ../accel/kvm/kvm-all.c:3205 #3 0x0000654c45e88964 in kvm_vcpu_thread_fn (arg=0x654c767adc20) at ../accel/kvm/kvm-accel-ops.c:51 #4 0x0000654c460bfdac in qemu_thread_start (args=0x654c767b8430) at ../util/qemu-thread-posix.c:393 #5 0x000078e0ddaa1e2e in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447 #6 0x000078e0ddb33a4c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 Thread 3 (Thread 0x78e0ce6ca6c0 (LWP 341223) "qemu-system-x86"): #0 __GI___ioctl (fd=16, request=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36 #1 0x0000654c45e85982 in kvm_vcpu_ioctl (cpu=0x654c767e2d60, type=44672) at ../accel/kvm/kvm-all.c:3403 #2 0x0000654c45e8502b in kvm_cpu_exec (cpu=0x654c767e2d60) at ../accel/kvm/kvm-all.c:3205 #3 0x0000654c45e88964 in kvm_vcpu_thread_fn (arg=0x654c767e2d60) at ../accel/kvm/kvm-accel-ops.c:51 #4 0x0000654c460bfdac in qemu_thread_start (args=0x654c767ec840) at ../util/qemu-thread-posix.c:393 #5 0x000078e0ddaa1e2e in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447 #6 0x000078e0ddb33a4c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 Thread 2 (Thread 0x78e0ce6ca6c0 (LWP 341226) "kvm-nx-lpage-re"): #0 0x0000000000000000 in ??? () Thread 1 (Thread 0x78e0d1b17640 (LWP 341220) "qemu-system-x86"): #0 0x000078e0ddb256a0 in __GI_ppoll (fds=0x654c76855080, nfds=9, timeout=<optimized out>, sigmask=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:42 #1 0x0000654c460e0c85 in qemu_poll_ns (fds=0x654c76855080, nfds=9, timeout=983442) at ../util/qemu-timer.c:342 #2 0x0000654c460db6a8 in os_host_main_loop_wait (timeout=983442) at ../util/main-loop.c:305 #3 0x0000654c460db804 in main_loop_wait (nonblocking=0) at ../util/main-loop.c:589 #4 0x0000654c45b77bf7 in qemu_main_loop () at ../system/runstate.c:905 #5 0x0000654c45fe7742 in qemu_default_main (opaque=0x0) at ../system/main.c:50 #6 0x0000654c45fe7800 in main (argc=24, argv=0x7fff678d1cf8) at ../system/main.c:93 ``` * After encountering the issue and subsequent attempt to power off the machine via `SIGINT`: ``` Thread 5 (Thread 0x78e0cf9ff6c0 (LWP 341221) "qemu-system-x86"): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000654c460c132b in qemu_futex_wait (f=0x654c47449618 <rcu_call_ready_event>, val=4294967295) at /home/zentarim/src/qemu/include/qemu/futex.h:47 #2 0x0000654c460c155f in qemu_event_wait (ev=0x654c47449618 <rcu_call_ready_event>) at ../util/event.c:162 #3 0x0000654c460cd230 in call_rcu_thread (opaque=0x0) at ../util/rcu.c:278 #4 0x0000654c460bfdac in qemu_thread_start (args=0x654c7647f760) at ../util/qemu-thread-posix.c:393 #5 0x000078e0ddaa1e2e in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447 #6 0x000078e0ddb33a4c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 Thread 4 (Thread 0x78e0ceecb6c0 (LWP 341222) "qemu-system-x86"): #0 0x000078e0dda9dffd in __futex_abstimed_wait_common64 (private=0, futex_word=0x654c7677f9ec, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (futex_word=0x654c7677f9ec, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x654c7677f9ec, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x000078e0ddaa0b6d in __pthread_cond_wait_common (cond=0x654c7677f9c0, mutex=0x654c474246e0 <bql>, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_wait (cond=0x654c7677f9c0, mutex=0x654c474246e0 <bql>) at ./nptl/pthread_cond_wait.c:627 #5 0x0000654c460bf735 in qemu_cond_wait_impl (cond=0x654c7677f9c0, mutex=0x654c474246e0 <bql>, file=0x654c462331dc "../system/cpus.c", line=465) at ../util/qemu-thread-posix.c:225 #6 0x0000654c45b475f2 in qemu_wait_io_event (cpu=0x654c767adc20) at ../system/cpus.c:465 #7 0x0000654c45e88988 in kvm_vcpu_thread_fn (arg=0x654c767adc20) at ../accel/kvm/kvm-accel-ops.c:56 #8 0x0000654c460bfdac in qemu_thread_start (args=0x654c767b8430) at ../util/qemu-thread-posix.c:393 #9 0x000078e0ddaa1e2e in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447 #10 0x000078e0ddb33a4c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 Thread 3 (Thread 0x78e0ce6ca6c0 (LWP 341223) "qemu-system-x86"): #0 futex_wait (futex_word=0x654c474246e0 <bql>, expected=2, private=0) at ../sysdeps/nptl/futex-internal.h:146 #1 __GI___lll_lock_wait (futex=futex@entry=0x654c474246e0 <bql>, private=0) at ./nptl/lowlevellock.c:49 #2 0x000078e0ddaa4742 in __pthread_mutex_cond_lock (mutex=mutex@entry=0x654c474246e0 <bql>) at ../nptl/pthread_mutex_lock.c:93 #3 0x000078e0ddaa0a50 in __pthread_cond_wait_common (cond=0x654c767b79f0, mutex=0x654c474246e0 <bql>, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:616 #4 ___pthread_cond_wait (cond=0x654c767b79f0, mutex=0x654c474246e0 <bql>) at ./nptl/pthread_cond_wait.c:627 #5 0x0000654c460bf735 in qemu_cond_wait_impl (cond=0x654c767b79f0, mutex=0x654c474246e0 <bql>, file=0x654c462331dc "../system/cpus.c", line=465) at ../util/qemu-thread-posix.c:225 #6 0x0000654c45b475f2 in qemu_wait_io_event (cpu=0x654c767e2d60) at ../system/cpus.c:465 #7 0x0000654c45e88988 in kvm_vcpu_thread_fn (arg=0x654c767e2d60) at ../accel/kvm/kvm-accel-ops.c:56 #8 0x0000654c460bfdac in qemu_thread_start (args=0x654c767ec840) at ../util/qemu-thread-posix.c:393 #9 0x000078e0ddaa1e2e in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447 #10 0x000078e0ddb33a4c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 Thread 2 (Thread 0x78e0ce6ca6c0 (LWP 341226) "kvm-nx-lpage-re"): #0 0x0000000000000000 in ??? () Thread 1 (Thread 0x78e0d1b17640 (LWP 341220) "qemu-system-x86"): #0 0x000078e0ddb256a0 in __GI_ppoll (fds=0x654c7754f110, nfds=1, timeout=<optimized out>, sigmask=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:42 #1 0x0000654c460e0c85 in qemu_poll_ns (fds=0x654c7754f110, nfds=1, timeout=999947113) at ../util/qemu-timer.c:342 #2 0x0000654c460bb480 in fdmon_poll_wait (ctx=0x654c764fa510, ready_list=0x7fff678d1908, timeout=999947113) at ../util/fdmon-poll.c:79 #3 0x0000654c460bae78 in aio_poll (ctx=0x654c764fa510, blocking=true) at ../util/aio-posix.c:730 #4 0x0000654c45f2b9d6 in bdrv_drain_all_begin () at ../block/io.c:531 #5 0x0000654c45f1aa4f in blk_drain_all () at ../block/block-backend.c:2099 #6 0x0000654c45a51b8b in virtio_scsi_dataplane_stop (vdev=0x654c773d08b0) at ../hw/scsi/virtio-scsi-dataplane.c:266 #7 0x0000654c45ad9b5f in virtio_bus_stop_ioeventfd (bus=0x654c773d0830) at ../hw/virtio/virtio-bus.c:259 #8 0x0000654c45adc80b in virtio_pci_stop_ioeventfd (proxy=0x654c773c82f0) at ../hw/virtio/virtio-pci.c:379 #9 0x0000654c45adef40 in virtio_pci_vmstate_change (d=0x654c773c82f0, running=false) at ../hw/virtio/virtio-pci.c:1366 #10 0x0000654c45e52c7a in virtio_vmstate_change (opaque=0x654c773d08b0, running=false, state=RUN_STATE_SHUTDOWN) at ../hw/virtio/virtio.c:3460 #11 0x0000654c45b76da3 in vm_state_notify (running=false, state=RUN_STATE_SHUTDOWN) at ../system/runstate.c:399 #12 0x0000654c45b470a6 in do_vm_stop (state=RUN_STATE_SHUTDOWN, send_stop=false) at ../system/cpus.c:300 #13 0x0000654c45b470eb in vm_shutdown () at ../system/cpus.c:322 #14 0x0000654c45b77dfb in qemu_cleanup (status=0) at ../system/runstate.c:981 #15 0x0000654c45fe774f in qemu_default_main (opaque=0x0) at ../system/main.c:51 #16 0x0000654c45fe7800 in main (argc=24, argv=0x7fff678d1cf8) at ../system/main.c:93 ```