Go garbage collector crashes when using qemu-x86_64 on an aarch64 host

Host environment

• Operating system: Alpine Linux
• OS/kernel version: 6.11-rc6
• Architecture: aarch64
• QEMU flavor: qemu-x86_64 (statically linked)
• QEMU version: from 9.0.1
• QEMU command line: using qemu via binfmt, see steps to reproduce below

Emulated/Virtualized environment

• Operating system: n/a
• OS/kernel version: n/a
• Architecture: x86_64

Description of problem

Apps compiled for Go and the Go compiler/tool itself crash when they are run with qemu-x86_64 on an AARCH64 host system. This was not a problem on QEMU 8.2.x (I bisected, see further down). I also seem to recall that Go 1.21 is fine on QEMU 9.x, so maybe some recent change in Go 1.22 + recent changes in QEMU broke something?

The crash from Go seems to be in the garbage collector, I cannot reproduce the issue when I disable the GC with GOGC=off.

Output from Go when it crashes:

$ sudo chroot . go build main.go
runtime: lfstack.push invalid packing: node=0xffff6542b2c0 cnt=0x1 packed=0xffff6542b2c00001 -> node=0xffffffff6542b2c0
fatal error: lfstack.push

runtime stack:
runtime.throw({0xa95b29?, 0x797b1e2a383c?})
        runtime/panic.go:1023 +0x5c fp=0xc000515f08 sp=0xc000515ed8 pc=0x43c27c
runtime.(*lfstack).push(0x0?, 0xc0005041c0?)
        runtime/lfstack.go:29 +0x125 fp=0xc000515f48 sp=0xc000515f08 pc=0x40fd45
runtime.(*spanSetBlockAlloc).free(...)
        runtime/mspanset.go:322
runtime.(*spanSet).reset(0xf46980)
        runtime/mspanset.go:264 +0x79 fp=0xc000515f78 sp=0xc000515f48 pc=0x437219
runtime.finishsweep_m()
        runtime/mgcsweep.go:258 +0x8d fp=0xc000515fb8 sp=0xc000515f78 pc=0x42a6cd
runtime.gcStart.func2()
        runtime/mgc.go:685 +0xf fp=0xc000515fc8 sp=0xc000515fb8 pc=0x46e40f
runtime.systemstack(0x0)
        runtime/asm_amd64.s:509 +0x4a fp=0xc000515fd8 sp=0xc000515fc8 pc=0x47442a

Steps to reproduce

0. Use an aarch64 host system!

1. Set up binfmt to use qemu-x86_64:

$ cat /proc/sys/fs/binfmt_misc/qemu-x86_64
enabled
interpreter /usr/bin/qemu-x86_64
flags: OCF
offset 0
magic 7f454c4602010100000000000000000002003e00
mask fffffffffffefe00fffffffffffffffffeffffff

2. Download/extract x86_64 rootfs:

$ curl -O https://dl-cdn.alpinelinux.org/alpine/v3.20/releases/x86_64/alpine-minirootfs-3.20.2-x86_64.tar.gz	

3. Create example app in the x86_64 rootfs:

package main

func main() {
}

4. Build using chroot:

$ sudo chroot /path/to/x86_64/rootfs apk add go
$ sudo chroot /path/to/x86_64/rootfs go build main.go
runtime: lfstack.push invalid packing: node=0xffff6542b2c0 cnt=0x1 packed=0xffff6542b2c00001 -> node=0xffffffff6542b2c0
fatal error: lfstack.push
...

5. As noted previously, if the Go garbage collector is disabled, then it works, presumably because it avoids the bug(?) in QEMU:

$ sudo chroot . env GOGC=off go build main.go
# might have to mount /dev to build successfully, but Go doesn't panic!

Additional information

I've bisected this exact crash/failure to:

commit 2952b642a555207748dd961fcbfdc48f198eebb6
Author: Richard Henderson <richard.henderson@linaro.org>
Date:   Tue Feb 13 10:20:27 2024 -1000

    linux-user: Split out do_munmap

    Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
    Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

Though a different crash starts happening at the commit before that one:

commit ad87d26e6bb13257409f412224c862fc54025e8b
Author: Richard Henderson <richard.henderson@linaro.org>
Date:   Tue Jan 2 12:57:55 2024 +1100

    linux-user: Do early mmap placement only for reserved_va

    For reserved_va, place all non-fixed maps then proceed
    as for MAP_FIXED.

    Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

FYI @rth7680