Fix kubernetes-helm-runner-incept image repository and make it fail when the runner doesn't start

kubernetes-helm-runner-incept reported success while its runner pod sat in ImagePullBackOff for the whole 120s window: https://gitlab.com/gitlab-org/ci-cd/tests/runner-incept/-/jobs/15985719873

Two things were wrong.

The job pulled from the wrong repository

It hardcoded gitlab-runner-dev / gitlab-runner-helper-dev. Upstream only publishes there from merge request pipelines (development docker images, gated on .rules:release:development:merge-requests); default-branch and tag pipelines push the same SHA tags to the canonical gitlab-org/gitlab-runner and ../gitlab-runner-helper. That job was triggered from tag v19.3.0, so the images existed — gitlab-org/gitlab-runner:alpine-9cbf0074 resolves, gitlab-runner-dev:alpine-9cbf0074 404s. So this has been broken for every tag and default-branch trigger, not just this one.

The tag was never the problem: make print_image_tags always emits the short SHA, so all three release jobs publish it. Only the repository varies.

The job now takes UPSTREAM_RUNNER_IMAGE_REPO / UPSTREAM_RUNNER_HELPER_IMAGE_REPO from the trigger, falling back to deriving them from a refs/merge-requests/ prefix on UPSTREAM_CI_COMMIT_REF when they aren't passed. A follow-up MR against gitlab-org/gitlab-runner will make the trigger pass them, after which the fallback can go — the repository choice is upstream's release policy and this project shouldn't be re-deriving it. Verified against the registry: MR-pipeline SHAs resolve in the dev repositories, main/tag/rc SHAs in the canonical ones.

The log now prints the full repository:tag. It previously printed only the tag, which was the one part that was already correct.

The job could not fail when the runner didn't come up

helm install has no --wait, and poll_k8s_cluster.sh printed kubectl get pods on a timer and always exited 0. It now waits on condition=Ready and exits non-zero, dumping describe and events on the way out — kubectl get pods alone can't distinguish a missing tag from a denied pull, and diagnosing the original failure needed an out-of-band registry probe.

Two details worth a look:

  • The wait selects -l app=gitlab-runner, and has to. manifests/helm_default_values.yaml sets useJobNamespace: true, which is not a key in the chart (it's runners.jobNamespace), so Helm ignores it and build pods share the default namespace with the runner manager. Unselected, the check would flap against transient build pods.
  • It keeps polling for the full window after the pod is ready instead of returning early. kubernetes-helm-hello-world has needs: [] and can only be picked up while the runner is alive, so exiting early would starve it.

Not addressed here

helm delete stays in script, so it is skipped on the new failure path. That doesn't regress runner cleanup: unregisterRunners: true renders a container preStop hook, which never fires when the container never started, and with terminationGracePeriodSeconds: 0 likely doesn't fire on success either. That's #33, with the scheduled cleanup-stale-runners job as the current backstop.

Merge request reports

Loading
Loading