When using podman (4.2.0) as runner, prebuilt images installed by gitlab-runner (15.8.0) rpm are not imported correctly

It's possible that this hasn't been noticed, since jobs will still run successfully unless the runner is unable to access the docker repository at registry.gitlab.com

Reproducing the problem

On runner correctly configured for podman usage:

$ su - gitlab-runner
$ podman images
REPOSITORY  TAG         IMAGE ID    CREATED     SIZE
$

When running a job the job log contains the following:

Pulling docker image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-12335144 ...
WARNING: Failed to pull image with policy "always": initializing source docker://registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-12335144: pinging container registry registry.gitlab.com: Get "https://registry.gitlab.com/v2/": dial tcp 35.227.35.254:443: i/o timeout (manager.go:237:243s)

This failed due to the fact the runner is in a walled garden and cannot access registry.gitlab.com

The debug log from the gitlab-runner service shows more detail of what is going on:

Jan 27 15:18:32 HOST_REDACTED gitlab-runner[8482]: Looking for prebuilt image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-12335144...  job=174616 project=1883 runner=JgGpA3Wu
Jan 27 15:18:32 HOST_REDACTED gitlab-runner[8482]: Failed to load prebuilt image from: /usr/bin/out/helper-images/prebuilt-alpine-x86_64.tar.xz error: open /usr/bin/out/helper-images/prebuilt-alpine-x86_64.tar.xz: no such file or directory  job=174616 project=1883 runner=JgGpA3Wu
Jan 27 15:18:32 HOST_REDACTED gitlab-runner[8482]: Failed to load prebuilt image from: /usr/helper-images/prebuilt-alpine-x86_64.tar.xz error: open /usr/helper-images/prebuilt-alpine-x86_64.tar.xz: no such file or directory  job=174616 project=1883 runner=JgGpA3Wu
Jan 27 15:18:32 HOST_REDACTED gitlab-runner[8482]: Failed to load prebuilt image from: /out/helper-images/prebuilt-alpine-x86_64.tar.xz error: open /out/helper-images/prebuilt-alpine-x86_64.tar.xz: no such file or directory job=174616 project=1883 runner=JgGpA3Wu
Jan 27 15:18:32 HOST_REDACTED gitlab-runner[8482]: Loading prebuilt image...                       job=174616 project=1883 runner=JgGpA3Wu
Jan 27 15:18:41 HOST_REDACTED gitlab-runner[8482]: Inspecting imported image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper failed: Error: No such image: registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-12335144 (docker.go:186:0s)  job=174616 project=1883 runner=JgGpA3Wu
Jan 27 15:18:41 HOST_REDACTED gitlab-runner[8482]: Failed to load prebuilt image from: /usr/lib/gitlab-runner/helper-images/prebuilt-alpine-x86_64.tar.xz error: Error: No such image: registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-12335144 (docker.go:186:0s)  job=174616 project=1883 runner=JgGpA3Wu
Jan 27 15:18:41 HOST_REDACTED gitlab-runner[8482]: Loading image form registry: registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-12335144  job=174616 project=1883 runner=JgGpA3Wu

This shows that the prebuild image is found in the RPM-installed location, and imported to podman, but cannot then be found. Checking podmans images again shows that the image has been loaded, but tagged "latest" not "x86_64-12335144":

$ podman images
REPOSITORY                                                         TAG         IMAGE ID      CREATED        SIZE
registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper  latest      2090f06e4fa2  4 minutes ago  69.3 MB

Workarounds

There are a few ways to work around the issue:

1 - specify an alternative helper_image location that the runner can access in config.toml

2 - Manually import the prebuilt image to podman prior to job execution

$ export RUNNER_VERSION=$(gitlab-runner --version | grep revision | cut -f 3 -d ' ')
$ podman import /usr/lib/gitlab-runner/helper-images/prebuilt-alpine-x86_64.tar.xz registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-${RUNNER_VERSION}
$ podman images
REPOSITORY                                                             TAG              IMAGE ID      CREATED        SIZE
registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper      x86_64-12335144  f6f858e7f79c  8 days ago     69.3 MB

3 - After a failed job, tag the loaded image with the correct tag.

4 - Ensure the runner can reach registry.gitlab.com

Options 1&4 have a downside that the first time the runner executes a job it will need to fetch the helper image. Option 3 is undesirable because it requires a failed job to run before it is fixed. Option 2 would need to be replicated for all the types needed - but at least avoids failures and additional downloads.

Proposal for solution to solve bug

{placeholder pending analysis and refinement}