Update CI runner image
Production Change
Change Summary
This change is to update our shared runner image to implement https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10782. This has been on staging since last week with no issues.
Change Details
- Services Impacted - CI
- Change Technician - @ahanselka
- Change Criticality - C4
- Change Type - changescheduled
- Change Reviewer - @cindy
- Due Date - 2020-09-22 19:00 UTC
- Time tracking - 60 minutes
- Downtime Component - No downtime necessary
Detailed steps for the change
Change Steps - steps to take to execute the change
Estimated Time to Complete (mins) - 20 minutes
-
Disable chef on all runners knife ssh "roles:gitlab-runner-srm" "sudo systemctl stop chef-client"
-
Apply this MR to change the image. -
Run Chef on shared-runners-manager-3
to ensure the change has the desired effect. -
Run Chef on all the other runners. knife ssh "roles:gitlab-runner-srm" "sudo chef-client"
Post-Change Steps - steps to take to verify the change
Estimated Time to Complete (mins) - 10 minutes
-
Ensure jobs that use the shared runners are still working.
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (mins) - 20
-
Revert the above MR
Monitoring
Key metrics to observe
- Metric: Jobs running on runners owned by GitLab Inc. (by runner type
- Location: https://dashboards.gitlab.net/d/000000159/ci?viewPanel=1&orgId=1&refresh=5m
- What changes to this metric should prompt a rollback: A sharp decline in jobs for
shared-runners-manager
Summary of infrastruture changes
-
Does this change introduce new compute instances? -
Does this change re-size any existing compute instances? -
Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?
Summary of the above
Changes checklist
-
This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled). -
This issue has the change technician as the assignee. -
Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. -
Necessary approvals have been completed based on the Change Management Workflow. -
Change has been tested in staging and resultes noted in a comment on this issue. -
A dry-run has been conducted and results noted in a comment on this issue. -
SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall
and this issue.) -
There are currently no active incidents.
Edited by Alex Hanselka