Connection refused after 8e1c21d0
After the upgrade to 1.11.0 from 1.10.1 in Arch Linux, I started seeing a "Connection refused" error when attempting to mount or access the shared folder in a couple of guest operating systems (Arch Linux, Fedora 40).
My virtiofsd command is:
doas $vfsd_bld/release/virtiofsd \
--cache always \
--shared-dir $VM_FOLDER/shared \
--socket-group $(id -gn) \
--socket-path $VM_FOLDER/vfsd.sock
and my QEMU command is:
qemu-system-x86_64 \
-display none \
-serial mon:stdio \
-nic user,model=virtio-net-pci,hostfwd=tcp::8022-:22 \
-object rng-random,filename=/dev/urandom,id=rng0 \
-device virtio-rng-pci \
-chardev socket,id=char0,path=$VM_FOLDER/vfsd.sock \
-device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=host \
-object memory-backend-memfd,id=mem,share=on,size=4G \
-numa node,memdev=mem \
-m 4G \
-device virtio-balloon \
-smp 2 \
-drive if=pflash,format=raw,file=$VM_FOLDER/efi.img,readonly=on \
-drive if=pflash,format=raw,file=$VM_FOLDER/efi_vars.img \
-cpu host \
-enable-kvm \
-M q35 \
-drive if=virtio,format=qcow2,file=$VM_FOLDER/disk.img
In Fedora 40, I see the following error when mounting and line in dmesg in the guest:
$ sudo mount -t virtiofs host /mnt/host
mount: /mnt/host: fsconfig system call failed: Connection refused.
dmesg(1) may have more information after failed mount system call.
$ sudo dmesg | grep virtiofs
[ 1.407510] virtiofs virtio2: virtio_fs_setup_dax: No cache capability
[ 40.681913] SELinux: (dev virtiofs, type virtiofs) getxattr errno 111
In Arch Linux, mounting appears to succeed but I get "Connection refused" when attempting to access the shared folder:
$ ls /mnt/host
$ sudo mount -t virtiofs host /mnt/host
$ ls /mnt/host
"/mnt/host": Connection refused (os error 111)
$ ls /mnt
[/mnt/host: Connection refused (os error 111)]
I was able to bisect this to commit 8e1c21d0.
# bad: [83057321d7920ab04c324981318ad62ee1a6d986] Merge branch 'release-1.11.0' into 'main'
# good: [3988b7304ceb2fdb4eed2c8bf8682e6ea19c4ecc] Merge branch 'release-1.10.1' into 'main'
git bisect start 'v1.11.0' 'v1.10.1'
# good: [d6c4f3955606a89ddbcdb5190b87ebd9bbc40570] opt: Introduce --allow-mmap flag
git bisect good d6c4f3955606a89ddbcdb5190b87ebd9bbc40570
# bad: [093c156a8fb04e62de4ab3c68330c8727b62e17f] Introduce --migration-verify-handles
git bisect bad 093c156a8fb04e62de4ab3c68330c8727b62e17f
# good: [859d0f0b469c67102e2528500d694ce65f43096c] Remove `&Inode` indirection
git bisect good 859d0f0b469c67102e2528500d694ce65f43096c
# bad: [cd2ca3b0a0c12fa0f2483acbe0e180f07e878eac] Add ReadDir::new_no_seek()
git bisect bad cd2ca3b0a0c12fa0f2483acbe0e180f07e878eac
# bad: [24d4352a06ddc834784d83f5507bbad6508d790c] Add type for strong inode references
git bisect bad 24d4352a06ddc834784d83f5507bbad6508d790c
# bad: [8e1c21d0121eea2659818253df59bb9c2180a971] Hide InodeStore locking
git bisect bad 8e1c21d0121eea2659818253df59bb9c2180a971
# first bad commit: [8e1c21d0121eea2659818253df59bb9c2180a971] Hide InodeStore locking
At the direct parent of that change (859d0f0b):
$ ls /mnt/host
$ sudo mount -t virtiofs host /mnt/host
$ ls /mnt/host
foo
There are no differences in the logs from the good and bad case:
[2024-06-17T21:49:33Z WARN virtiofsd::limits] Failure when trying to set the limit to 1000000, the hard limit (524288) of open file descriptors is used instead.
[2024-06-17T21:49:33Z INFO virtiofsd] Waiting for vhost-user socket connection...
[2024-06-17T21:49:40Z INFO virtiofsd] Client connected, servicing requests
Warning: Cannot announce submounts, client does not support it
[2024-06-17T21:52:25Z INFO virtiofsd] Client disconnected, shutting down
In case it matters, I am using QEMU emulator version 9.0.1 and the guest kernels are Linux 6.9.y. I am more than happy to provide any additional information or test patches as necessary.