exec user process caused: exec format error

Summary

Executing binaries from a different architecture via binfmt+qemu recently (1-2 weeks) started to fail on some runners. So far this seems to happen on runner XxUrkriXT but others could be affected as well.

My hypothesis is that this is a new runner that was deployed without noticing that some packages are missing respect to older runners.

Steps to reproduce

Try with executing anything in image arm32v7/alpine:3.15 on a Linux amd64 runner.

Example Project

Look, this is our project: https://gitlab.com/pygolo/py.

The relevant section of .gitlab-ci.yml is:

alpine:
  extends:
    - .apk
    - .test
  image: $ARCH/alpine:$VERSION
  parallel:
    matrix:
      - VERSION: "3.19"
        ARCH: amd64
      - VERSION: "3.18"
        ARCH: amd64
      - VERSION: "3.17"
        ARCH: ppc64le
      - VERSION: "3.16"
        ARCH: arm64v8
      - VERSION: "3.15"
        ARCH: arm32v7

What is the current bug behavior?

The execution stops with an error as soon as the kernel tries to execute the binary compiled for the other architecture (https://gitlab.com/pygolo/py/-/jobs/6688250512):

Using docker image sha256:1634fbc5edbc74cb5105c89b8d8ca36420a328b38d32e523da0c308f8b2c9020 for arm32v7/alpine:3.15 with digest arm32v7/alpine@sha256:840178c78cc7f3ca793ec097e1f7f16b66f413d6b55650c7efa2f9878322f5f8 ...
standard_init_linux.go:228: exec user process caused: exec format error

What is the expected correct behavior?

If binfmt tools and qemu are correctly installed, as they've been so far in the SaaS Linux runners, execution of some other architecture's binary is totally transparent. This works also across containers and indeed it's something we used to test the Pygolo project also on less common architectures like arm32v7 and ppc64le, aside from the more common amd64 and arm64, by just specifying the architecture of the container image.

Look at this other successful execution on zxwgkjAPH (https://gitlab.com/pygolo/py/-/jobs/6688264967):

Using docker image sha256:1634fbc5edbc74cb5105c89b8d8ca36420a328b38d32e523da0c308f8b2c9020 for arm32v7/alpine:3.15 with digest arm32v7/alpine@sha256:840178c78cc7f3ca793ec097e1f7f16b66f413d6b55650c7efa2f9878322f5f8 ...
Running on runner-zxwgkjap-project-43840705-concurrent-0 via runner-zxwgkjap-s-l-s-amd64-1713804136-3fee0e31...
Fetching changes with git depth set to 20...

Relevant logs and/or screenshots

Linked above.

Possible fixes

Ensure that these packages are installed on the runner: binfmt-support and qemu-user-binfmt.

Edited by Domenico Andreoli