[production] Enable of `load_balancing_for_build_hooks_worker`
Production Change
Change Summary
BuildHooksWorker
should now utilize new Sidekiq read-only database replicas capabilities, and hit the replica most of the time, since we set data consistency for this worker to delayed
- The rollout issue: gitlab-org/gitlab#326095 (closed)
- The previous incident caused by this rollout: #4336 (closed)
Change Details
- Services Impacted - ServiceSidekiq ServicePatroni ServicePostgres
- Change Technician - @nmilojevic1
- Change Criticality - C2
- Change Type - changescheduled
- Change Reviewer - @ahmadsherif
- Due Date - 10 May 2021, 10:00 UTC
- Time tracking - 45m
- Downtime Component - No downtime expected
Detailed steps for the change
Incremental rollout. Proposed increments are: 10%
, 50%
, 100%
.
Proposed minimum time between increments is 15 minutes.
Pre-Change Steps - steps to be completed before execution of the change
Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes
-
Set label changein-progress on this issue
Change Steps - steps to take to execute the change
Estimated Time to Complete (mins) - 45m
-
/chatops run feature set load_balancing_for_build_hooks_worker 10
-
/chatops run feature set load_balancing_for_build_hooks_worker 50
-
/chatops run feature set load_balancing_for_build_hooks_worker 100
Post-Change Steps - steps to take to verify the change
Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes
-
Review with EOC
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (mins) - 1m
-
/chatops run feature set load_balancing_for_build_hooks_worker false
Monitoring
- Metric: Sidekiq queue detail dashboard
- Metric: Postgres Async (Sidekiq) replica Connection Pool Utilization per Node
- Metric: Queue lenghts
- Metric: Postgres RSS Memory usage
- Metric: Sidekiq connection pool saturation details
- Metric: Log
Key metrics to observe
-
Metric: Log
- Location: https://log.gprd.gitlab.net/goto/6b54e091862b1fb63eb5e89fc42fe2f0
- There should not be a lot of host_offline events in the logs
-
Metric: Queue lenghts
- Location: https://dashboards.gitlab.net/d/sidekiq-shard-detail/sidekiq-shard-detail?viewPanel=3&orgId=1&from=now-1h&to=now
- Queue size should not increase
Summary of infrastructure changes
-
Does this change introduce new compute instances? -
Does this change re-size any existing compute instances? -
Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?
Summary of the above
Changes checklist
-
This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities. -
This issue has the change technician as the assignee. -
Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. -
Necessary approvals have been completed based on the Change Management Workflow. -
Change has been tested in staging and results noted in a comment on this issue. -
A dry-run has been conducted and results noted in a comment on this issue. -
SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall
and this issue and await their acknowledgement.) -
There are currently no active incidents.
Edited by Nikola Milojevic