Cloud native build logs - 25% gitlab.com rollout

Production Change

Change Summary

We are working on rollout the cloud native build logs feature to production.

• This effort is described in an architectural blueprint ➡ https://docs.gitlab.com/ee/architecture/blueprints/cloud_native_build_logs/
• Rollout issue ➡ gitlab-org/gitlab#241471 (closed)

Notes:

• This feature has been successfully enabled in gitlab-org/gitlab a few weeks ago, and is working fine there
• This feature has been successfully enabled in gitlab-com/www-gitlab-com project and behaves well there
• This feature has been rolled out at 10% on production during the last week

Change Details

1. Services Impacted - Redis, APIs
2. Change Technician - @grzesiek
3. Change Criticality - C2
4. Change Type - changeunscheduled
5. Change Reviewer - @hphilipps
6. Due Date - 2020-10-20
7. Time tracking - 4 hour
8. Downtime Component - No

Detailed steps for the change

Change Steps - steps to take to execute the change

Estimated Time to Complete (mins) - 4 hours

Incrementally increase the rollout up to 25%, wait 30-60 minutes before increments.

• /chatops run feature set ci_enable_live_trace 15 --actors
• /chatops run feature set ci_enable_live_trace 20 --actors
• /chatops run feature set ci_enable_live_trace 25 --actors

Post-Change Steps - steps to take to verify the change

Estimated Time to Complete (mins) - 30 minutes

• Check correctness of build traces using gitlab_ci_trace_operations_total Prometheus metric

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

Estimated Time to Complete (mins) - 1 minute

• /chatops run feature set ci_enable_live_trace 10 --actors

Monitoring

• Sentry errors containing "trace" keyword -> https://sentry.gitlab.net/gitlab/gitlabcom/?query=trace
• API dashboard for build status / trace operations - PUT /api/jobs/:id / PATCH /api/jobs/:id/trace
• Build details page -> GET trace.json / GET raw
• Redis memory -> Redis Overview Dashboard

Key metrics to observe

• Metric: gitlab_ci_trace_operations_total
  • Location: Prometheus
  • What changes to this metric should prompt a rollback: a lot of invalid traces counted.

Changes checklist

• This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled).
• This issue has the change technician as the assignee.
• Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed.
• Necessary approvals have been completed based on the Change Management Workflow.
• Change has been tested in staging and resultes noted in a comment on this issue.
• A dry-run has been conducted and results noted in a comment on this issue.
• SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall and this issue.)
• There are currently no active incidents.