GitLab Runner Pulling wrong helper image (which does not exist)

Summary

Starting with version 16.8.0, GitLab Runner attempts to pull a runner-helper image which doesn't appear to exist.

I am running GitLab 16.11 on EKS in a FIPS environment. I deploy both GitLab and GitLab Runner using helm. Recently, I attempted to upgrade GitLab Runner to 16.11 but found that pipelines would fail to start with the error.

registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:ubi-fips-v16.11: not found

I went through all the versions between where I was upgrading from and 16.11 and found that 16.8 is the earliest version where I would see this failure.

I then looked to see if there were any gitlab-runner-helper:ubi-fips-v${VERSION} images in the registry, and there are none. There are however gitlab-runner-helper:ubi-fips-${ARCHITECTURE}-v${VERSION} images.

It appears that when ubi-fips is the helper_image_flavor the architecture will not be affixed. However, I was curious and tried to set helper_image_flavor = "ubi-fips-x86_64" and observed it would try to pull registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:ubi-fips-x86_64-x86_64-v16.8.1. Not that in this the architecture would be included.

Steps to reproduce

.gitlab-ci.yml

```yaml
include:
  - template: "Workflows/MergeRequest-Pipelines.gitlab-ci.yml"

variables:
  DOCKERFILE: dockerfile
  KUBERNETES_MEMORY_REQUEST: "4Gi"
  KUBERNETES_MEMORY_LIMIT: "4Gi"
  KUBERNETES_CPU_REQUEST: "4"
  KUBERNETES_CPU_LIMIT: "4"

.build:
  image:
    name: gcr.io/kaniko-project/executor:v1.9.2-debug
  before_script:
    - mkdir -p ${VERSION}/certs/
    - cp /etc/gitlab-runner/certs/* ${VERSION}/certs/
  parallel:
    matrix:
      - VERSION: ["3.8", "3.10"]


build.dev:
  extends: .build
  script:
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}/${VERSION}"
      --dockerfile "${CI_PROJECT_DIR}/${VERSION}/${DOCKERFILE}"
      --destination "${CI_REGISTRY_IMAGE}:${VERSION}-${CI_COMMIT_SHORT_SHA}-dev"
      --cache=true
  rules:
    - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
      when: never
    - when: on_success

build.prod:
  extends: .build
  script:
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}/${VERSION}"
      --dockerfile "${CI_PROJECT_DIR}/${VERSION}/${DOCKERFILE}"
      --destination "${CI_REGISTRY_IMAGE}:${VERSION}-${CI_COMMIT_SHORT_SHA}"
      --cache=true

  rules:
    - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
      when: on_success
    - when: never

</details>

## Actual behavior



## Expected behavior

The runner-helper image should be pulled

## Relevant logs and/or screenshots


<details>
<summary> job log </summary>

```sh
Running with gitlab-runner 16.8.1 (a6097117)
  on gitlab-runner-65f56459f4-4v2dq sBeEw5uw, system ID: r_sMwSgX1xuzCF
Resolving secrets 00:00
Preparing the "kubernetes" executor 00:00
Using Kubernetes namespace: platform-gitlab-runners
Using Kubernetes executor with image gcr.io/kaniko-project/executor:v1.9.2-debug ...
Using attach strategy to execute scripts...
Preparing environment 00:08
Using FF_USE_POD_ACTIVE_DEADLINE_SECONDS, the Pod activeDeadlineSeconds will be set to the job timeout: 1h0m0s...
Waiting for pod platform-gitlab-runners/runner-sbeew5uw-project-195-concurrent-0-r8nrhofa to be running, status is Pending
WARNING: Failed to pull image with policy "": image pull failed: Back-off pulling image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:ubi-fips-v16.8.1"
ERROR: Job failed: prepare environment: waiting for pod running: pulling image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:ubi-fips-v16.8.1": image pull failed: Back-off pulling image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:ubi-fips-v16.8.1". Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

Environment description

config.toml contents

[[runners]]
  [runners.cache]
    Type   = "s3"
    Shared = true
    [runners.cache.s3]
      ServerAddress      = "s3-fips.${cache_aws_s3_bucket_region}.amazonaws.com"
      BucketName         = "${cache_aws_s3_bucket_name}"
      BucketLocation     = "${cache_aws_s3_bucket_region}"
      Insecure           = false
      AuthenticationType = "iam"
  [runners.kubernetes]
    image = "alpine:3"

    poll_interval = 5
    poll_timeout  = 300

    allow_privilege_escalation = false
    privileged                 = false

    cpu_limit      = "500m"
    cpu_request    = "250m"
    memory_limit   = "500Mi"
    memory_request = "500Mi"

    helper_cpu_limit      = "500m"
    helper_cpu_request    = "250m"
    helper_memory_limit   = "500Mi"
    helper_memory_request = "500Mi"

    service_cpu_limit      = "500m"
    service_cpu_request    = "250m"
    service_memory_limit   = "500Mi"
    service_memory_request = "500Mi"

    cpu_limit_overwrite_max_allowed      = "8000m"
    cpu_request_overwrite_max_allowed    = "8000m"
    memory_limit_overwrite_max_allowed   = "16Gi"
    memory_request_overwrite_max_allowed = "16Gi"

    helper_cpu_limit_overwrite_max_allowed      = "8000m"
    helper_cpu_request_overwrite_max_allowed    = "8000m"
    helper_memory_limit_overwrite_max_allowed   = "16Gi"
    helper_memory_request_overwrite_max_allowed = "16Gi"

    service_cpu_limit_overwrite_max_allowed      = "8000m"
    service_cpu_request_overwrite_max_allowed    = "8000m"
    service_memory_limit_overwrite_max_allowed   = "16Gi"
    service_memory_request_overwrite_max_allowed = "16Gi"

    cleanup_grace_period_seconds         = 30
    pod_termination_grace_period_seconds = 300

    [runners.kubernetes.node_selector]
      "pipeline.devops.company.com/node" = "general"

    [runners.kubernetes.node_tolerations]
      "pipeline.devops.company.com/node" = "NoSchedule"

    [[runners.kubernetes.volumes.secret]]
      name = "gitlab-runners-ca-certs"
      mount_path = "/etc/gitlab-runner/certs/"

Used GitLab Runner version

Running with gitlab-runner 16.8.1 (a6097117)
  on gitlab-runner-65f56459f4-4v2dq sBeEw5uw, system ID: r_sMwSgX1xuzCF
Preparing the "kubernetes" executor 00:00
Using Kubernetes namespace: platform-gitlab-runners
Using Kubernetes executor with image gcr.io/kaniko-project/executor:v1.9.2-debug ...

I have observed this all from 16.8.0 - 16.11

Possible fixes