GitLab Runner build promotion on stable branches
Summary
The initial issue that raised the incident was that a subset of tags were missing for the helper images. This caused CI/CD pipeline failures across GitLab repositories, and anyone trying to pull registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v17.8.0 which is an image used by GitLab Runner 17.8.0 to run every job.
We do not current have a way to test a GitLab Runner release before releasing it. This corrective action adds the ability to test image releases before releasing them by just "promoting" a tag we have already built and tested.
See #38344 (comment 2258629197)
Related Incident
- Incident: 2025-01-16: GitLab-runner image v17.8.0 not found (gitlab-com/gl-infra/production#19129 - closed) • Sarah Walker
- Review: Incident Review: GitLab Runner Helper v17.8 Ima... (gitlab-com/gl-infra/production#19131 - closed) • Arran Walker, Romuald Atchadé - OOO until Jan 11th, 2026
Desired Outcome/Acceptance Criteria
- The final step in releasing GitLab Runner images is "promoting" a tagged image.
- All building and testing of the candidate image(s) happens before promotion.
Associated Services
ServiceCI Runners in GitLab.com / GitLab Infrastructure Team / Production Engineering
Corrective Action Issue Checklist
-
Link the incident(s) this corrective action arose from -
Give context for what problem this corrective action is trying to prevent re-occurring -
Assign a severity label (this is the highest sev of related incidents, defaults to 'severity::4') -
Assign a priority (this will default to 'Production Engineering::P4' but should match the severity of the related incident) -
Assign a service label -
Assign a team label
Edited by Joe Burnett