Fix kubernetes-helm-runner-incept image repository and make it fail when the runner doesn't start
kubernetes-helm-runner-incept reported success while its runner pod sat in
ImagePullBackOff for the whole 120s window:
https://gitlab.com/gitlab-org/ci-cd/tests/runner-incept/-/jobs/15985719873
Two things were wrong.
The job pulled from the wrong repository
It hardcoded gitlab-runner-dev / gitlab-runner-helper-dev. Upstream only publishes
there from merge request pipelines (development docker images, gated on
.rules:release:development:merge-requests); default-branch and tag pipelines push the
same SHA tags to the canonical gitlab-org/gitlab-runner and ../gitlab-runner-helper.
That job was triggered from tag v19.3.0, so the images existed —
gitlab-org/gitlab-runner:alpine-9cbf0074 resolves, gitlab-runner-dev:alpine-9cbf0074
404s. So this has been broken for every tag and default-branch trigger, not just this one.
The tag was never the problem: make print_image_tags always emits the short SHA, so all
three release jobs publish it. Only the repository varies.
The job now takes UPSTREAM_RUNNER_IMAGE_REPO / UPSTREAM_RUNNER_HELPER_IMAGE_REPO from
the trigger, falling back to deriving them from a refs/merge-requests/ prefix on
UPSTREAM_CI_COMMIT_REF when they aren't passed. A follow-up MR against
gitlab-org/gitlab-runner will make the trigger pass them, after which the fallback can
go — the repository choice is upstream's release policy and this project shouldn't be
re-deriving it. Verified against the registry: MR-pipeline SHAs resolve in the dev
repositories, main/tag/rc SHAs in the canonical ones.
The log now prints the full repository:tag. It previously printed only the tag, which
was the one part that was already correct.
The job could not fail when the runner didn't come up
helm install has no --wait, and poll_k8s_cluster.sh printed kubectl get pods on a
timer and always exited 0. It now waits on condition=Ready and exits non-zero, dumping
describe and events on the way out — kubectl get pods alone can't distinguish a
missing tag from a denied pull, and diagnosing the original failure needed an out-of-band
registry probe.
Two details worth a look:
- The wait selects
-l app=gitlab-runner, and has to.manifests/helm_default_values.yamlsetsuseJobNamespace: true, which is not a key in the chart (it'srunners.jobNamespace), so Helm ignores it and build pods share thedefaultnamespace with the runner manager. Unselected, the check would flap against transient build pods. - It keeps polling for the full window after the pod is ready instead of returning early.
kubernetes-helm-hello-worldhasneeds: []and can only be picked up while the runner is alive, so exiting early would starve it.
Not addressed here
helm delete stays in script, so it is skipped on the new failure path. That doesn't
regress runner cleanup: unregisterRunners: true renders a container preStop hook, which
never fires when the container never started, and with terminationGracePeriodSeconds: 0
likely doesn't fire on success either. That's #33, with the scheduled
cleanup-stale-runners job as the current backstop.