virtio-gpu with venus support fails with a non-helpful error when virgl_render_server is not available

Host environment

• Operating system: Debian Trixie
• OS/kernel version: 6.15.1
• Architecture: x86_64
• QEMU flavor: qemu-system-x86_64, qemu-system-aarch64
• QEMU version: QEMU emulator version 10.0.50 (v10.0.0-1541-g640bb21a8e-dirty) but also in 10.0
• QEMU command line:

# This is running the func-aarch64-aarch64_virt_gpu directly
./qemu-system-aarch64 -machine virt -serial mon:stdio -accel tcg -cpu cortex-a72 -machine virt,gic-version=max \
      -kernel /home/alex/.cache/qemu/download/7888c51c55d37e86bbbdeb5acea9f08c34e6b0f03c1f5b2463285f6a6f6eec8b \
      -append "printk.time=0 console=ttyAMA0 root=/dev/vda" \
      -smp 2 -m 2048 \
      -device virtio-gpu-gl-pci,hostmem=4G,blob=on,venus=on \
      -display gtk,gl=on \
      -device virtio-blk-device,drive=hd0 \
      -blockdev driver=raw,file.driver=file,node-name=hd0,read-only=on,file.filename=/home/alex/lsrc/qemu.git/builds/all/tests/functional/aarch64/test_aarch64_virt_gpu.Aarch64VirtGPUMachine.test_aarch64_virt_with_vulkan_gpu/scratch/d45118c899420b7e673f1539a37a35480134b3e36e3a59e2cb69b1781cbb14ef -snapshot \

Emulated/Virtualized environment

• Operating system: Linux
• OS/kernel version: 6.12.16
• Architecture: Arm and x86_64

Description of problem

When starting up QEMU with venus=on and -display gtk,gl=on but without virgl_render_server installed (usually found in /usr/libexec/virgl_render_server, Debian packages it up separately from libvirglrender) you will get weird errors when you try and check the status of Vulkan in the guest:

# vulkaninfo --summary                                                                                                                                                        
virtio_gpu_virgl_process_cmd: ctrl 0x10c, error 0x1200                                                                                                                        
virtio_gpu_virgl_process_cmd: ctrl 0x208, error 0x1203                                                                                                                        
[drm:virtio_gpu_dequeue_ctrl_func] *ERROR* response 0x1200 (command 0x10c)             
[drm:virtio_gpu_dequeue_ctrl_func] *ERROR* response 0x1203 (command 0x208)             
virtio_gpu_virgl_process_cmd: ctrl 0x209, error 0x1203                                                                                                                        
virtio_gpu_virgl_process_cmd: ctrl 0x102, error 0x1203                                 
[drm:virtio_gpu_dequeue_ctrl_func] *ERROR* response 0x1203 (command 0x209)             
[drm:virtio_gpu_dequeue_ctrl_func] *ERROR* response 0x1203 (command 0x102)                                                                                                    
ERROR at ./vulkaninfo.h:576:vkCreateInstance failed with ERROR_OUT_OF_HOST_MEMORY      
# QEMU 10.0.50 monitor - type 'help' for more information

What is especially confusing is the exact same guest images work run through the functional tests:

  ./pyvenv/bin/meson test  --setup thorough func-aarch64-aarch64_virt_gpu
ninja: Entering directory `/home/alex/lsrc/qemu.git/builds/all'
[1/6] Generating qemu-version.h with a custom command (wrapped by meson to capture output)
1/1 qemu:func-thorough+func-aarch64-thorough+thorough / func-aarch64-aarch64_virt_gpu        OK              66.70s   3 subtests passed

Ok:                 1   
Expected Fail:      0   
Fail:               0   
Unexpected Pass:    0   
Skipped:            0   
Timeout:            0   

Full log written to /home/alex/lsrc/qemu.git/builds/all/meson-logs/testlog-thorough.txt

The reason it seems is when run with a headless mode (-display dbus,gl=on) the underlying libvirglrenderer doesn't need to start virgl_render_server.

Additionally when run under ASAN we get the following scary error:

🕙14:47:29 alex@draig:qemu.git/builds/virtio-gpu  on  HEAD (640bb21) (REBASING 1/5) [$!?] took 4s [🔴 ERROR]
✗  env LD_PRELOAD=/lib/x86_64-linux-gnu/libasan.so.8:/home/alex/lsrc/qemu.git/builds/extra.libs/virglrenderer.git/build/src/libvirglrenderer.so ASAN_OPTIONS=halt_on_error=1 ./qemu-system-x86_64 \
                                              -machine type=q35,accel=kvm \
                                              -cpu host \
                                              -smp 4 \
                                              -device virtio-net-pci,netdev=unet \
                                              -netdev user,id=unet,hostfwd=tcp::2222-:22 \
                                              -serial mon:stdio \
                                              -drive file=/home/alex/lsrc/tests/buildroot.git/builds/x86_64/images/rootfs.ext2,if=virtio,format=raw -snapshot \
                                              -kernel /home/alex/lsrc/tests/buildroot.git/builds/x86_64/images/bzImage  \
                                              -append "root=/dev/vda" \
                                              -m 24G \
                                              -object memory-backend-memfd,id=mem,size=24G,share=on \
                                              -device virtio-vga-gl,hostmem=4G,blob=on,venus=on \
                                              -display gtk,gl=on,show-cursor=off \
                                              -device virtio-tablet-pci -device virtio-keyboard-pci
==393560==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
AddressSanitizer:DEADLYSIGNAL
=================================================================
==393589==ERROR: AddressSanitizer: SEGV on unknown address 0x7fe361936990 (pc 0x7fef951ae637 bp 0x7ffe1cf02320 sp 0x7ffe1cf02260 T0)
==393589==The signal is caused by a READ memory access.
    #0 0x7fef951ae637 in __pthread_clockjoin_ex nptl/pthread_join_common.c:43
    #1 0x7fef994ece45 in operator() ../../../../src/libsanitizer/asan/asan_interceptors.cpp:288
    #2 0x7fef994ece45 in Join<___interceptor_pthread_join(void*, void**)::<lambda()> > ../../../../src/libsanitizer/sanitizer_common/sanitizer_thread_arg_retval.h:75
    #3 0x7fef994ece45 in pthread_join ../../../../src/libsanitizer/asan/asan_interceptors.cpp:287
    #4 0x7fe377bafda1  (/lib/x86_64-linux-gnu/libgallium-25.0.5-1.so+0x5afda1) (BuildId: 17b621a998972f5828194cce760f15e26f24192a)
    #5 0x7fe377b7afc5  (/lib/x86_64-linux-gnu/libgallium-25.0.5-1.so+0x57afc5) (BuildId: 17b621a998972f5828194cce760f15e26f24192a)
    #6 0x7fe377b7b05b  (/lib/x86_64-linux-gnu/libgallium-25.0.5-1.so+0x57b05b) (BuildId: 17b621a998972f5828194cce760f15e26f24192a)
    #7 0x7fef9515c290 in __run_exit_handlers stdlib/exit.c:118
    #8 0x7fef9515c359 in __GI_exit stdlib/exit.c:148
    #9 0x7fef99284d5c in proxy_server_create (/home/alex/lsrc/qemu.git/builds/extra.libs/virglrenderer.git/build/src/libvirglrenderer.so+0x61d5c) (BuildId: 45e734a7b7242a549d892bd231d09d13688c9fdf)
    #10 0x7fef99284a62 in proxy_renderer_init (/home/alex/lsrc/qemu.git/builds/extra.libs/virglrenderer.git/build/src/libvirglrenderer.so+0x61a62) (BuildId: 45e734a7b7242a549d892bd231d09d13688c9fdf)
    #11 0x7fef9923ff31 in virgl_renderer_init (/home/alex/lsrc/qemu.git/builds/extra.libs/virglrenderer.git/build/src/libvirglrenderer.so+0x1cf31) (BuildId: 45e734a7b7242a549d892bd231d09d13688c9fdf)
    #12 0x5586d8eef5da in virtio_gpu_virgl_init (/home/alex/lsrc/qemu.git/builds/virtio-gpu/qemu-system-x86_64+0x154f5da) (BuildId: 43294584552f22c254b8f5e0c3ae41f702ff1c3f)
    #13 0x5586d8ee28bf in virtio_gpu_gl_handle_ctrl (/home/alex/lsrc/qemu.git/builds/virtio-gpu/qemu-system-x86_64+0x15428bf) (BuildId: 43294584552f22c254b8f5e0c3ae41f702ff1c3f)
    #14 0x5586d9703bad in aio_bh_call (/home/alex/lsrc/qemu.git/builds/virtio-gpu/qemu-system-x86_64+0x1d63bad) (BuildId: 43294584552f22c254b8f5e0c3ae41f702ff1c3f)
    #15 0x5586d9704126 in aio_bh_poll (/home/alex/lsrc/qemu.git/builds/virtio-gpu/qemu-system-x86_64+0x1d64126) (BuildId: 43294584552f22c254b8f5e0c3ae41f702ff1c3f)
    #16 0x5586d96ca17d in aio_dispatch (/home/alex/lsrc/qemu.git/builds/virtio-gpu/qemu-system-x86_64+0x1d2a17d) (BuildId: 43294584552f22c254b8f5e0c3ae41f702ff1c3f)
    #17 0x5586d97036f1 in aio_ctx_dispatch (/home/alex/lsrc/qemu.git/builds/virtio-gpu/qemu-system-x86_64+0x1d636f1) (BuildId: 43294584552f22c254b8f5e0c3ae41f702ff1c3f)
    #18 0x7fef97e363c4  (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x5c3c4) (BuildId: 575d4368412866d4b8d9cd3260838f71a2795bbc)
    #19 0x7fef97e38cb7 in g_main_context_dispatch (/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x5ecb7) (BuildId: 575d4368412866d4b8d9cd3260838f71a2795bbc)
    #20 0x5586d97070a7 in main_loop_wait (/home/alex/lsrc/qemu.git/builds/virtio-gpu/qemu-system-x86_64+0x1d670a7) (BuildId: 43294584552f22c254b8f5e0c3ae41f702ff1c3f)
    #21 0x5586d8cb26c2 in qemu_main_loop (/home/alex/lsrc/qemu.git/builds/virtio-gpu/qemu-system-x86_64+0x13126c2) (BuildId: 43294584552f22c254b8f5e0c3ae41f702ff1c3f)
    #22 0x5586d953eecb in qemu_default_main (/home/alex/lsrc/qemu.git/builds/virtio-gpu/qemu-system-x86_64+0x1b9eecb) (BuildId: 43294584552f22c254b8f5e0c3ae41f702ff1c3f)
    #23 0x5586d850c79b in main (/home/alex/lsrc/qemu.git/builds/virtio-gpu/qemu-system-x86_64+0xb6c79b) (BuildId: 43294584552f22c254b8f5e0c3ae41f702ff1c3f)
    #24 0x7fef95143ca7 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #25 0x7fef95143d64 in __libc_start_main_impl ../csu/libc-start.c:360
    #26 0x5586d8511ba0 in _start (/home/alex/lsrc/qemu.git/builds/virtio-gpu/qemu-system-x86_64+0xb71ba0) (BuildId: 43294584552f22c254b8f5e0c3ae41f702ff1c3f)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV nptl/pthread_join_common.c:43 in __pthread_clockjoin_ex
==393589==ABORTING

Which implies there is some sort of failure in the fail path which should probably be looked at. However the main issue is QEMU currently has no good way of detecting this failed creation of the proxy server because the initialisation happens in a different thread from the main start-up so virgl_renderer_init() can return a success despite things falling over once we try and probe the status of Vulkan.