Hang with 14.6.0 docker executor and podman

Summary

On a RHEL or CentOS 8 system with gitlab-runner 14.6.0 and rootless podman, CI jobs hang after starting the container.

Downgrading to gitlab-runner 14.5.2 resolves the issue.

Steps to reproduce

Configure a docker runner with rootless podman, which roughly looks like this:

loginctl enable-linger gitlab-runner
su - gitlab-runner -c 'XDG_RUNTIME_DIR=/run/user/$( id -u ) systemctl --user enable --now podman.socket'
mkdir -p /etc/systemd/system/gitlab-runner.service.d
echo -e "[Service]\nEnvironment=\"DOCKER_HOST=unix:///run/user/$( id -u gitlab-runner )/podman/podman.sock\"" > /etc/systemd/system/gitlab-runner.service.d/podman.conf

Actual behavior

Jobs do not start. In the job log, the output looks like this:

Running with gitlab-runner 14.6.0 (5316d4ac)
  on runner01d-docker xxXXyyYY
Preparing the "docker" executor
Using Docker executor with image docker.io/library/ruby:2.7.2 ...
Pulling docker image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-5316d4ac ...
Using docker image sha256:f5a8ee0df245ace063e3c4aab58fddcac7e8809fa9458eb64e66355dc1c4f0ad for registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-5316d4ac with digest registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper@sha256:c238a26be8b4a50516da6a02399a8706899e4ed0b9b4e77af01c04a16a2bc1dd ...
Pulling docker image docker.io/library/ruby:2.7.2 ...
Using docker image sha256:e6c92ed2f03be9788b80944e148783bef8e7d0fa8d9755b62e9f03429e85a327 for docker.io/library/ruby:2.7.2 with digest docker.io/library/ruby@sha256:1dd0106849233fcd913b7c4608078fa1a53a5e3ce1af2a55e4d726b0d8868e2f ...
Preparing environment
Running on runner-zvazmjfp-project-29-concurrent-0 via runner01d.gitlab.domain...

It stops there until the job times out.

Expected behavior

With 14.5.2, jobs continue past that point.

Running with gitlab-runner 14.5.2 (e91107dd)
  on runner01c-docker aaAAbbBB
Preparing the "docker" executor
Using Docker executor with image python:alpine ...
Pulling docker image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-e91107dd ...
Using docker image sha256:f8d786201f794725aa1bd4fb89b14ed576c60fa9546e15162dbb8587310d5828 for registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-e91107dd with digest registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper@sha256:b9aff524500e7aaa9851959b076aad70cd63b8a4e182bde69615bd36aaeaecc0 ...
Pulling docker image python:alpine ...
Using docker image sha256:eb5bc7d10d52904953df1e5f3460f4f4154b7911603a3889a35ffcb9b6e75b81 for python:alpine with digest docker.io/library/python@sha256:affe0faa14e7553fc570beec3864e74b5e36f8c19b2bb49ae8ba79c0e9e7236e ...
Preparing environment
Running on runner-yerfev72-project-29-concurrent-0 via runner01c.gitlab.domain...
Getting source from Git repository
Fetching changes...
[...]

Environment description

These are all on self-hosted runners and GitLab.

The bug has been seen on RHEL 8 with podman 3.2.3 and 3.3.1 and on CentOS 8 with podman 3.2.3. Various containers were tested.

config.toml contents

concurrent = 2
[[runners]]
executor = "docker"
name = "runner01d-docker"
token = "..."
url = "https://gitlab.gitlab.domain/"
[runners.docker]
image = "ubuntu:trusty"

Used GitLab Runner version

Version:      14.6.0
Git revision: 5316d4ac
Git branch:   14-6-stable
GO version:   go1.13.8
Built:        2021-12-17T17:36:04+0000
OS/Arch:      linux/amd64

Possible fixes

dnf downgrade gitlab-runner-14.5.2 works around the issue.