Cloud native build logs - 60% gitlab.com rollout
Production Change
Change Summary
We are working on rollout the cloud native build logs feature to production.
- This effort is described in an architectural blueprint
➡ https://docs.gitlab.com/ee/architecture/blueprints/cloud_native_build_logs/ - Rollout issue
➡ gitlab-org/gitlab#241471 (closed)
Notes:
- This feature has been successfully enabled in
gitlab-org/gitlab
a few weeks ago, and is working fine there - This feature has been successfully enabled in
gitlab-com/www-gitlab-com
project and behaves well there - This feature has been rolled out at 25% on production lately
Change Details
- Services Impacted - Redis, APIs
- Change Technician - @grzesiek
- Change Criticality - C2
- Change Type - changeunscheduled
- Change Reviewer - @hphilipps
- Due Date - 2020-10-22
- Time tracking - 8 hour
- Downtime Component - No
Detailed steps for the change
Change Steps - steps to take to execute the change
Estimated Time to Complete (mins) - 8 hours
Incrementally increase the rollout from 25% up to 60%, wait 30-60 minutes before increments.
-
/chatops run feature set ci_enable_live_trace 30 --actors
-
/chatops run feature set ci_enable_live_trace 40 --actors
-
/chatops run feature set ci_enable_live_trace 50 --actors
-
/chatops run feature set ci_enable_live_trace 60 --actors
Post-Change Steps - steps to take to verify the change
Estimated Time to Complete (mins) - 1 hour
-
Check correctness of build traces using gitlab_ci_trace_operations_total
Prometheus metric
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (mins) - 1 minute
-
/chatops run feature set ci_enable_live_trace 25 --actors
Monitoring
- Sentry errors containing "trace" keyword -> https://sentry.gitlab.net/gitlab/gitlabcom/?query=trace
- API dashboard for build status / trace operations - PUT /api/jobs/:id / PATCH /api/jobs/:id/trace
- Build details page -> GET trace.json / GET raw
- Redis memory -> Redis Overview Dashboard
Key metrics to observe
- Metric:
gitlab_ci_trace_operations_total
- Location: Prometheus
- What changes to this metric should prompt a rollback: a lot of
invalid
traces counted.
Changes checklist
-
This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled). -
This issue has the change technician as the assignee. -
Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. -
Necessary approvals have been completed based on the Change Management Workflow. -
Change has been tested in staging and resultes noted in a comment on this issue. -
A dry-run has been conducted and results noted in a comment on this issue. -
SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall
and this issue.) -
There are currently no active incidents.
Edited by Grzegorz Bizon