Os boot issues on 9p filesystem due to unix domain sockets open failure

Since our last production update of Ubuntu 22.04 we see all of our virtualised container workloads failing with:

qemu-system-s390x: 9p: broken or compromised client detected; attempt to open special file (i.e. neither regular file, nor directory)

reported in the host logs and no log output from the workloads themselves.
We use runq based container virtualisation which uses 9p filesystem to share the workload through qemu into the VM.

Bug analysis

The containers fail because syslog cannot open its unix domain socket on the filesystem.
We tracked the change that provokes this error to a CVE change in qemu that forbids opening of special files to prevent exposing data from the host. Special files should be handled by the guest os.
Unix domain socket files are also special files, and they are handled by the guest OS in their entirety, and the 9p server in qemu assigns them individual inodes so they are safe to open. But they must be opened so their fd can be passed to the appropriate connect() or bind() function so the OS can use them.
Socket files don't have a traditional read or write functionality, they are mere representatives for a local address.
There is no convention for where domain socket files should go, so there is no easy fix by just creating a tmpfs somewhere.
We also see other workloads and services failing for not being able to open their local socket files.

Host environment

• Operating system: Ubuntu 22.04 LTS
• OS/kernel version: 5.15.0-92-generic #102-Ubuntu SMP Wed Jan 10 09:35:24 UTC 2024 s390x s390x s390x GNU/Linux
• Architecture: s390x ( but any is affected)
• QEMU flavor: qemu-system-s390x ( but any is affected)
• QEMU version: QEMU emulator version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.16)
• QEMU command line:

   qemu-system-s390x \
  -m 8192 \
  -smp 4 \
  -nodefaults -nographic -no-reboot -no-user-config \
  -kernel ./target/boot/vmlinuz \
  -initrd ./target/boot/initrd.img \
  -append 'root=fsRoot rw rootfstype=9p rootflags=trans=virtio,version=9p2000.L,msize=512000,cache=mmap,posixacl console=ttysclp0 init=/test_init quiet' \
  -fsdev local,security_model=passthrough,multidevs=remap,id=fsdev-fsRoot,path=./target \
  -device virtio-9p-pci,id=fsRoot,fsdev=fsdev-fsRoot,mount_tag=fsRoot \
  -device virtio-serial-ccw -device sclpconsole,chardev=console \
  -chardev stdio,id=console,signal=off

Emulated/Virtualized environment

• Operating system: Ubuntu (23.11)
• OS/kernel version: Any
• Architecture: s390x

Description of problem

Unix filesystem API is broken, unix domain socket special files return an error at open()

Steps to reproduce

Simple script. Tries to use netcat to get data through a local unix domain socket file

#!/bin/bash

# Cleanup target dir
[ -d ./target ] && rm -rf target
mkdir target

# Add configuration updates
mkdir -p ./target/etc/initramfs-tools/
echo 9p >> ./target/etc/initramfs-tools/modules
echo 9pnet_virtio >> ./target/etc/initramfs-tools/modules

# Add the test script
cat > ./target/test_init << EOF
#!/bin/bash

echo "Test for unix domain sockets"

nc -Ul /socket &
sleep 1
echo "Sockets work" | nc -UN /socket || echo "Sockets fail"

echo o > /proc/sysrq-trigger
sleep 999
EOF
chmod 700 ./target/test_init

# Create an Ubuntu 23.10 around it
echo "Creating Ubuntu target OS"
debootstrap --variant=minbase\
            --include=udev,kmod,initramfs-tools,systemd,netcat-openbsd,linux-image-generic \
            --exclude=man,bash-completion \
            mantic ./target > /dev/null || exit 1

# Run the test in 9p forwarded filesystem
echo "Running OS in qemu"
qemu-system-s390x \
  -m 8192 \
  -smp 4 \
  -nodefaults -nographic -no-reboot -no-user-config \
  -kernel ./target/boot/vmlinuz \
  -initrd ./target/boot/initrd.img \
  -append 'root=fsRoot rw rootfstype=9p rootflags=trans=virtio,version=9p2000.L,msize=512000,cache=mmap,posixacl console=ttysclp0 init=/test_init quiet' \
  -fsdev local,security_model=passthrough,multidevs=remap,id=fsdev-fsRoot,path=./target \
  -device virtio-9p-pci,id=fsRoot,fsdev=fsdev-fsRoot,mount_tag=fsRoot \
  -device virtio-serial-ccw -device sclpconsole,chardev=console \
  -chardev stdio,id=console,signal=off 

Additional information

Test output:

Test for unix domain sockets
qemu-system-s390x: 9p: broken or compromised client detected; attempt to open special file (i.e. neither regular file, nor directory)
nc: No such device or address
nc: /socket: No such file or directory
Sockets fail