macOS M1 runners hang after `script` step in Gitaly and cause concurrent jobs to hang as well
We're currently in the process of migrating to the M1 runners in the Gitaly project. Despite some performance issues we have been working around in ci: Speed up macOS job (gitlab-org/gitaly!5890 - merged), we also see that the runner reliably hangs after the script
step.
An example job of this can be found in the Gitaly project. What is most interesting is that the hang also caused other, concurrent M1 jobs in a different pipeline to hang in the middle of the step they were handling. Logs can be found here. It is easy to see the point in time where the job switched between doing something and hanging based on the log volume. It is seemingly busy-spinning trying to submit the job to the coordinator, until it then finally times out with:
Error while executing file based variables removal script
The problem is consistent and happens on ever CI run.