Enable cloud native build logs in `gitlab-org/gitlab`.

Production Change

Change Summary

We are working on rollout the cloud native build logs feature to production.

• This effort is described in an architectural blueprint ➡ https://docs.gitlab.com/ee/architecture/blueprints/cloud_native_build_logs/
• Rollout issue ➡ gitlab-org/gitlab#241471 (closed)

Note This feature had been successfully enabled for gitlab-org/gitlab a few weeks ago - we've build additional verification mechanisms and want to re-enable it ➡ #2501 (closed)

Change Details

1. Services Impacted - Redis, APIs
2. Change Technician - @grzesiek
3. Change Criticality - C3
4. Change Type - changeunscheduled
5. Change Reviewer - @ahmadsherif
6. Due Date - 2010.10.01
7. Time tracking - 1 hour
8. Downtime Component - No

Detailed steps for the change

Pre-Change Steps - steps to be completed before execution of the change

Estimated Time to Complete (mins) - 1 minute

• Write a message about the rollout in #whats-happening-at-gitlab @grzesiek

Change Steps - steps to take to execute the change

Estimated Time to Complete (mins) - 1 minute

• /chatops run feature set --project=gitlab-org/gitlab ci_enable_live_trace true
• /chatops run feature set --project=gitlab-org/gitlab ci_accept_trace true

Post-Change Steps - steps to take to verify the change

Estimated Time to Complete (mins) - 30 minutes

• Check correctness of build traces using gitlab_ci_trace_operations_total Prometheus metric

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

Estimated Time to Complete (mins) - 1 minute

• /chatops run feature set --project=gitlab-org/gitlab ci_enable_live_trace false
• /chatops run feature set --project=gitlab-org/gitlab ci_accept_trace false

Monitoring

• Sentry errors containing "trace" keyword -> https://sentry.gitlab.net/gitlab/gitlabcom/?query=trace
• API dashboard for build status / trace operations - PUT /api/jobs/:id / PATCH /api/jobs/:id/trace
• Build details page -> GET trace.json / GET raw
• Redis memory -> Redis Overview Dashboard

Key metrics to observe

• Metric: gitlab_ci_trace_operations_total
  • Location: Prometheus
  • What changes to this metric should prompt a rollback: a lot of invalid traces counted.

Changes checklist

• This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled).
• This issue has the change technician as the assignee.
• Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed.
• Necessary approvals have been completed based on the Change Management Workflow.
• Change has been tested in staging and resultes noted in a comment on this issue.
• A dry-run has been conducted and results noted in a comment on this issue.
• SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall and this issue.)
• There are currently no active incidents.