virtiofs mount hangs in guest after Fedora 39 upgrade
After upgrading to Fedora 39 yesterday (which switched virtiofsd from the C implementation to the Rust implementation), some of my virtiofs mounts have started to hang inside the guest. When this happens, any process in the guest VM that tries to access the mountpoint gets stuck in the sleep state. If I try to `umount -l <mountpoint> && mount <mountpoint>`, then the `mount` command similarly hangs forever.
The hang doesn't occur immediately after the guest boots. It'll work fine for a little while and then suddenly hang later. In my test VM, I've seen this happen on a virtiofs mountpoint that has no files open and is just idling.
Package versions:
* Both host and guest run Fedora 39 (x86_64)
* `kernel-6.5.8-300.fc39.x86_64`
* `libvirt-daemon-9.7.0-1.fc39.x86_64`
* `virtiofsd-1.7.0-5.fc39.x86_64`
Inside the guest, when the hang occurs, I don't see any logs at all--neither in `dmesg` nor `journalctl`. On the host, `/var/log/libvirt/qemu/svc-1-fs0-virtiofsd.log` contains:
<details>
<summary>(click to expand)</summary>
```
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 9
fv_queue_thread: Start for queue 1 kick_fd 12
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 9
fv_queue_thread: Start for queue 1 kick_fd 12
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 12
fv_queue_thread: Start for queue 0 kick_fd 9
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 9
fv_queue_thread: Start for queue 1 kick_fd 12
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=9
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=12
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 9
fv_queue_thread: Start for queue 1 kick_fd 12
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=9
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=12
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 9
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 12
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_thread: Start for queue 1 kick_fd 13
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 13
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 13
fv_queue_thread: Start for queue 0 kick_fd 10
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_thread: Start for queue 1 kick_fd 13
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_thread: Start for queue 1 kick_fd 13
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_thread: Start for queue 1 kick_fd 13
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_thread: Start for queue 1 kick_fd 13
virtio_loop: ppoll interrupted, going around
stop_all_queues: Stopping queue 0 thread
fv_queue_thread: kill event on queue 0 - quitting
stop_all_queues: Stopping queue 1 thread
fv_queue_thread: kill event on queue 1 - quitting
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 13
fv_queue_thread: Start for queue 0 kick_fd 10
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_thread: Start for queue 1 kick_fd 13
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=10
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=13
virtio_loop: Unexpected poll revents 11
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_thread: Start for queue 1 kick_fd 13
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=10
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=13
virtio_loop: Unexpected poll revents 11
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_thread: Start for queue 1 kick_fd 13
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=10
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=13
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 13
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=10
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=13
virtio_loop: Unexpected poll revents 11
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_thread: Start for queue 1 kick_fd 13
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=10
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=13
virtio_loop: Unexpected poll revents 11
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_thread: Start for queue 1 kick_fd 13
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=10
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=13
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 13
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=10
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=13
virtio_loop: Unexpected poll revents 11
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_thread: Start for queue 1 kick_fd 13
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=10
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=13
virtio_loop: Unexpected poll revents 11
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_thread: Start for queue 1 kick_fd 13
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=10
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=13
virtio_loop: Unexpected poll revents 11
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_thread: Start for queue 1 kick_fd 13
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=10
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=13
virtio_loop: Unexpected poll revents 11
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_thread: Start for queue 1 kick_fd 13
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=10
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=13
virtio_loop: Unexpected poll revents 11
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_thread: Start for queue 1 kick_fd 13
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=10
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=13
virtio_loop: Unexpected poll revents 11
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_thread: Start for queue 1 kick_fd 13
virtio_loop: ppoll interrupted, going around
stop_all_queues: Stopping queue 0 thread
fv_queue_thread: kill event on queue 0 - quitting
stop_all_queues: Stopping queue 1 thread
fv_queue_thread: kill event on queue 1 - quitting
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_thread: Start for queue 1 kick_fd 13
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=10
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=13
virtio_loop: Unexpected poll revents 11
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_thread: Start for queue 1 kick_fd 13
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=10
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=13
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_thread: Start for queue 1 kick_fd 13
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=10
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=13
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 13
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 13
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_thread: Start for queue 1 kick_fd 13
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 10
fv_queue_thread: Start for queue 1 kick_fd 13
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_thread: Start for queue 1 kick_fd 15
virtio_loop: ppoll interrupted, going around
stop_all_queues: Stopping queue 0 thread
fv_queue_thread: kill event on queue 0 - quitting
stop_all_queues: Stopping queue 1 thread
fv_queue_thread: kill event on queue 1 - quitting
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
virtio_loop: ppoll interrupted, going around
stop_all_queues: Stopping queue 0 thread
fv_queue_thread: kill event on queue 0 - quitting
stop_all_queues: Stopping queue 1 thread
fv_queue_thread: kill event on queue 1 - quitting
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=11
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=15
virtio_loop: Unexpected poll revents 11
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=11
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=15
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=11
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=15
virtio_loop: Unexpected poll revents 11
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=11
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=15
virtio_loop: Unexpected poll revents 11
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=11
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=15
virtio_loop: Unexpected poll revents 11
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=11
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=15
virtio_loop: Unexpected poll revents 11
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_thread: Start for queue 1 kick_fd 15
virtio_loop: ppoll interrupted, going around
stop_all_queues: Stopping queue 0 thread
fv_queue_thread: kill event on queue 0 - quitting
stop_all_queues: Stopping queue 1 thread
fv_queue_thread: kill event on queue 1 - quitting
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_thread: Start for queue 1 kick_fd 15
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=11
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=15
virtio_loop: Unexpected poll revents 11
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=11
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=15
virtio_loop: Unexpected poll revents 11
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=11
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=15
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=11
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=15
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=11
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=15
virtio_loop: Unexpected poll revents 11
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=11
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=15
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=11
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=15
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=11
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=15
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=11
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=15
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=11
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=15
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=11
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=15
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_thread: Start for queue 0 kick_fd 11
virtio_loop: ppoll interrupted, going around
stop_all_queues: Stopping queue 0 thread
fv_queue_thread: kill event on queue 0 - quitting
stop_all_queues: Stopping queue 1 thread
fv_queue_thread: kill event on queue 1 - quitting
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=11
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=15
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=11
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=15
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=11
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=15
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
virtio_loop: ppoll interrupted, going around
stop_all_queues: Stopping queue 0 thread
fv_queue_thread: kill event on queue 0 - quitting
stop_all_queues: Stopping queue 1 thread
fv_queue_thread: kill event on queue 1 - quitting
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_thread: Start for queue 1 kick_fd 15
virtio_loop: ppoll interrupted, going around
stop_all_queues: Stopping queue 0 thread
fv_queue_thread: kill event on queue 0 - quitting
stop_all_queues: Stopping queue 1 thread
fv_queue_thread: kill event on queue 1 - quitting
virtio_loop: Exit
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=11
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=15
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=11
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=15
fv_queue_set_started: qidx=0 started=1
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=11
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=15
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
fv_queue_set_started: qidx=0 started=0
fv_queue_thread: kill event on queue 0 - quitting
fv_remove_watch: TODO! fd=11
fv_queue_set_started: qidx=1 started=0
fv_queue_thread: kill event on queue 1 - quitting
fv_remove_watch: TODO! fd=15
fv_queue_set_started: qidx=0 started=1
fv_queue_thread: Start for queue 0 kick_fd 11
fv_queue_set_started: qidx=1 started=1
fv_queue_thread: Start for queue 1 kick_fd 15
virtio_loop: ppoll interrupted, going around
stop_all_queues: Stopping queue 0 thread
fv_queue_thread: kill event on queue 0 - quitting
stop_all_queues: Stopping queue 1 thread
fv_queue_thread: kill event on queue 1 - quitting
virtio_loop: Exit
[2023-10-24T23:53:36Z WARN virtiofsd] Use of deprecated option format '-o': Please specify options without it (e.g., '--cache auto' instead of '-o cache=auto')
[2023-10-24T23:53:36Z INFO virtiofsd] Waiting for vhost-user socket connection...
[2023-10-24T23:53:36Z INFO virtiofsd] Client connected, servicing requests
```
</details>
The virtiofsd command is launched by libvirt + qemu:
```
root 4537 0.0 0.0 6168 3840 ? S Oct24 0:00 /usr/libexec/virtiofsd --fd=46 -o source=/satapool0/enc/public
root 4541 0.0 0.0 8513096 10896 ? Sl Oct24 0:02 /usr/libexec/virtiofsd --fd=46 -o source=/satapool0/enc/public
```
and the volume is mounted in the guest's fstab with:
```
public /media/public virtiofs ro,context=system_u:object_r:httpd_sys_content_t:s0 0 0
```
The libvirt XML for virtiofs contains:
```xml
<filesystem type='mount' accessmode='passthrough'>
<driver type='virtiofs'/>
<binary path='/usr/libexec/virtiofsd'/>
<source dir='/satapool0/enc/public'/>
<target dir='public'/>
<alias name='fs0'/>
<address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
</filesystem>
```
Given the lack of relevant log messages (at least that I can see), I'm not sure how to troubleshoot this. I'm happy to test anything or grab additional information if needed. I currently have the VM left in the hanging state.
issue