GitLab Runner Pulling wrong helper image (which does not exist)
Summary
Starting with version 16.8.0, GitLab Runner attempts to pull a runner-helper image which doesn't appear to exist.
I am running GitLab 16.11 on EKS in a FIPS environment. I deploy both GitLab and GitLab Runner using helm. Recently, I attempted to upgrade GitLab Runner to 16.11 but found that pipelines would fail to start with the error.
registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:ubi-fips-v16.11: not found
I went through all the versions between where I was upgrading from and 16.11 and found that 16.8 is the earliest version where I would see this failure.
I then looked to see if there were any gitlab-runner-helper:ubi-fips-v${VERSION} images in the registry, and there are none.
There are however gitlab-runner-helper:ubi-fips-${ARCHITECTURE}-v${VERSION} images.
It appears that when ubi-fips is the helper_image_flavor the architecture will not be affixed.
However, I was curious and tried to set helper_image_flavor = "ubi-fips-x86_64" and observed it would try to pull registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:ubi-fips-x86_64-x86_64-v16.8.1. Not that in this the architecture would be included.
Steps to reproduce
.gitlab-ci.yml
```yaml
include:
- template: "Workflows/MergeRequest-Pipelines.gitlab-ci.yml"
variables:
DOCKERFILE: dockerfile
KUBERNETES_MEMORY_REQUEST: "4Gi"
KUBERNETES_MEMORY_LIMIT: "4Gi"
KUBERNETES_CPU_REQUEST: "4"
KUBERNETES_CPU_LIMIT: "4"
.build:
image:
name: gcr.io/kaniko-project/executor:v1.9.2-debug
before_script:
- mkdir -p ${VERSION}/certs/
- cp /etc/gitlab-runner/certs/* ${VERSION}/certs/
parallel:
matrix:
- VERSION: ["3.8", "3.10"]
build.dev:
extends: .build
script:
- /kaniko/executor
--context "${CI_PROJECT_DIR}/${VERSION}"
--dockerfile "${CI_PROJECT_DIR}/${VERSION}/${DOCKERFILE}"
--destination "${CI_REGISTRY_IMAGE}:${VERSION}-${CI_COMMIT_SHORT_SHA}-dev"
--cache=true
rules:
- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
when: never
- when: on_success
build.prod:
extends: .build
script:
- /kaniko/executor
--context "${CI_PROJECT_DIR}/${VERSION}"
--dockerfile "${CI_PROJECT_DIR}/${VERSION}/${DOCKERFILE}"
--destination "${CI_REGISTRY_IMAGE}:${VERSION}-${CI_COMMIT_SHORT_SHA}"
--cache=true
rules:
- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
when: on_success
- when: never
</details>
## Actual behavior
## Expected behavior
The runner-helper image should be pulled
## Relevant logs and/or screenshots
<details>
<summary> job log </summary>
```sh
Running with gitlab-runner 16.8.1 (a6097117)
on gitlab-runner-65f56459f4-4v2dq sBeEw5uw, system ID: r_sMwSgX1xuzCF
Resolving secrets 00:00
Preparing the "kubernetes" executor 00:00
Using Kubernetes namespace: platform-gitlab-runners
Using Kubernetes executor with image gcr.io/kaniko-project/executor:v1.9.2-debug ...
Using attach strategy to execute scripts...
Preparing environment 00:08
Using FF_USE_POD_ACTIVE_DEADLINE_SECONDS, the Pod activeDeadlineSeconds will be set to the job timeout: 1h0m0s...
Waiting for pod platform-gitlab-runners/runner-sbeew5uw-project-195-concurrent-0-r8nrhofa to be running, status is Pending
WARNING: Failed to pull image with policy "": image pull failed: Back-off pulling image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:ubi-fips-v16.8.1"
ERROR: Job failed: prepare environment: waiting for pod running: pulling image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:ubi-fips-v16.8.1": image pull failed: Back-off pulling image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:ubi-fips-v16.8.1". Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more informationEnvironment description
config.toml contents
[[runners]]
[runners.cache]
Type = "s3"
Shared = true
[runners.cache.s3]
ServerAddress = "s3-fips.${cache_aws_s3_bucket_region}.amazonaws.com"
BucketName = "${cache_aws_s3_bucket_name}"
BucketLocation = "${cache_aws_s3_bucket_region}"
Insecure = false
AuthenticationType = "iam"
[runners.kubernetes]
image = "alpine:3"
poll_interval = 5
poll_timeout = 300
allow_privilege_escalation = false
privileged = false
cpu_limit = "500m"
cpu_request = "250m"
memory_limit = "500Mi"
memory_request = "500Mi"
helper_cpu_limit = "500m"
helper_cpu_request = "250m"
helper_memory_limit = "500Mi"
helper_memory_request = "500Mi"
service_cpu_limit = "500m"
service_cpu_request = "250m"
service_memory_limit = "500Mi"
service_memory_request = "500Mi"
cpu_limit_overwrite_max_allowed = "8000m"
cpu_request_overwrite_max_allowed = "8000m"
memory_limit_overwrite_max_allowed = "16Gi"
memory_request_overwrite_max_allowed = "16Gi"
helper_cpu_limit_overwrite_max_allowed = "8000m"
helper_cpu_request_overwrite_max_allowed = "8000m"
helper_memory_limit_overwrite_max_allowed = "16Gi"
helper_memory_request_overwrite_max_allowed = "16Gi"
service_cpu_limit_overwrite_max_allowed = "8000m"
service_cpu_request_overwrite_max_allowed = "8000m"
service_memory_limit_overwrite_max_allowed = "16Gi"
service_memory_request_overwrite_max_allowed = "16Gi"
cleanup_grace_period_seconds = 30
pod_termination_grace_period_seconds = 300
[runners.kubernetes.node_selector]
"pipeline.devops.company.com/node" = "general"
[runners.kubernetes.node_tolerations]
"pipeline.devops.company.com/node" = "NoSchedule"
[[runners.kubernetes.volumes.secret]]
name = "gitlab-runners-ca-certs"
mount_path = "/etc/gitlab-runner/certs/"Used GitLab Runner version
Running with gitlab-runner 16.8.1 (a6097117)
on gitlab-runner-65f56459f4-4v2dq sBeEw5uw, system ID: r_sMwSgX1xuzCF
Preparing the "kubernetes" executor 00:00
Using Kubernetes namespace: platform-gitlab-runners
Using Kubernetes executor with image gcr.io/kaniko-project/executor:v1.9.2-debug ...I have observed this all from 16.8.0 - 16.11