Skip to content

GitLab CI unable to pull non amd64 docker images (e.g. i386, arm)

Summary

As of last week, my CI job was able to use the i386/debian:bullseye image without any problem. As of today, it fails to pull the image. The only change made was that I upgraded my self-hosted instance to get the latest security patches. I use self-hosted GitLab as well as a self-hosted runner. The CI job was not modified, nor the config.toml. The only changes have been upgrades via apt.

Troubleshooting

I logged onto the runner and ran docker pull i386/debian:bullseye --platform=linux/i386 to confirm that the image can be downloaded from Docker Hub. It worked without any problems.

I tried running docker pull i386/debian:bullseye --platform=linux/arm64 and docker pull i386/debian:bullseye and both failed with the same warning message seen in the CI logs (see below).

As a temporary workaround, I tried specifying the image name by its SHA256 hash instead of by name. This did not work on the command line (without the --platform=i386 option), nor in the CI job. Downloading the image by hash worked fine when the --platform=i386 option was added. There are no known workarounds at this time.

Steps to reproduce

Try running a CI job that uses an i386 image. This reportedly can be reproduced with arm images on GitLab.com as well.

I am using GitLab v17.5.2-ee and gitlab-runner 16.9.1-1. Here's the output for gitlab-runner --version:

Version:      16.9.1
Git revision: 782c6ecb
Git branch:   16-9-stable
GO version:   go1.21.7
Built:        2024-02-28T16:51:21+0000
OS/Arch:      linux/amd64

I have not tried to downgrade my production instance to 17.5.1-ee to determine if that fixes this issue. If anyone has a test instance that is running 17.5.1-ee that they could try a CI job which uses a i386/debian image, it'd be great to see if it can be reproduced there. If not, it'd be even more interesting to upgrade that test instance to 17.5.2-ee to determine if the problem suddenly appears.

What is the current bug behavior?

GitLab runner is unable to pull non amd64 images.

What is the expected correct behavior?

GitLab runner can pull non amd64 images (as it could last week).

Relevant logs and/or screenshots

Here's the error from my logs:

Preparing the "docker" executor 00:18
Using Docker executor with image i386/debian:bullseye ...
Pulling docker image i386/debian:bullseye ...
WARNING: Failed to pull image with policy "always": no matching manifest for linux/amd64 in the manifest list entries (manager.go:250:0s)
ERROR: Job failed: failed to pull image "i386/debian:bullseye" with specified policies [always]: no matching manifest for linux/amd64 in the manifest list entries (manager.go:250:0s)

Possible fixes

If the CI runner checked out the docker image with the --platform option, and it were possible to set the value for that option in the .gitlab-ci.yml file, it would be possible to at least work around this issue.

This would mean developers would need to change definitions like

image: i386/debian:bullseye

to:

image:
  name: i386/debian:bullseye
  platform: linux/i386

That workaround is not as good as having GitLab CI work as it did previously, where no changes to the gitlab-ci.yml files were needed, however it would be nice to at least have some path forward to restore builds to working order. I looked in the documentation on images but the only property I could find for an image apart from the name was the entrypoint.

I'm marking this as a severity 1 because all non-amd64 builds are currently broken with no workaround.

I'm marking this as priority 1 because everyone's non-amd64 build processes will be broken for 60+ days (the minimum expected time to fix for any lower priority level) at any lower level. This would mean projects using GitLab could not release security patches with the official CI based build processes. Projects would need to have developers manually build i386 releases on their local machines to get security patches released.

If I'm incorrect about the impact or the lack of a workaround, or these levels are otherwise inappropriate, please feel free to adjust the priority/severity as appropriate.