Unblock gitlab org workflow by rebalancing issues relative positions
Production Change
Change Summary
Run issues relative position rebalancing for gitlab-org group based on gitlab-org/gitlab#276483 (comment 443077310)
The operation is run in a transaction so it would either succeed of rollback in case of a timeout or exception. If the change is successful it should help unblock gitlab-org users which currently encounter disfunction at issues level on boards.
Change Details
- Services Impacted - ServicePatroni
- Change Technician - @acroitor
- Change Criticality - C2
- Change Type - changeunscheduled
- Change Reviewer - @ahmadsherif
- Due Date - Date and time (in UTC) for the execution of the change
- Time tracking - Time, in minutes, needed to execute all change steps, including rollback
- Downtime Component - If there is a need for downtime, include downtime estimate here
Detailed steps for the change
Pre-Change Steps - steps to be completed before execution of the change
Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes
-
Rails console access -
Enable rebalance_issues
FF for gitlab-org -
Enable issue_rebalancing_optimization
FF -
Prepare a modified version of the IssueRebalancingService
to allow for a bigger number of issues
class IssueRebalancingService
MAX_ISSUE_COUNT = 200_000
end
...
Change Steps - steps to take to execute the change
Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes
-
Run IssueRebalancingService
- ~3 mins.IssueRebalancingService.new(Project.find(278964).issues.take).execute
-
on success of the step 1 also run IssuePlacementWorker
to place accumulated issues with null relative position to the end of the lists.
Post-Change Steps - steps to take to verify the change
Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes
-
Check there are no more issues with relative position = NULL on gitlab-org -
Check the min and max relative positions on gitlab-org -
Disable rebalance_issues
FF for gitlab-org -
Disable issue_rebalancing_optimization
FF
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes
-
Auto rollbacks in case of a query timeout as it is being run in a transaction. -
Disable rebalance_issues
FF for gitlab-org -
Disable issue_rebalancing_optimization
FF
Monitoring
Key metrics to observe
- Metric: Metric Name
- Location: Dashboard URL
- What changes to this metric should prompt a rollback: Describe Changes
Summary of infrastructure changes
-
Does this change introduce new compute instances? -
Does this change re-size any existing compute instances? -
Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?
Summary of the above
Changes checklist
-
This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities. -
This issue has the change technician as the assignee. -
Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. -
Necessary approvals have been completed based on the Change Management Workflow. -
Change has been tested in staging and results noted in a comment on this issue. -
A dry-run has been conducted and results noted in a comment on this issue. -
SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall
and this issue and await their acknowledgement.) -
There are currently no active incidents.
Edited by Nels Nelson