Unblock gitlab org workflow by rebalancing issues relative positions

Production Change

Change Summary

Run issues relative position rebalancing for gitlab-org group based on gitlab-org/gitlab#276483 (comment 443077310)

The operation is run in a transaction so it would either succeed of rollback in case of a timeout or exception. If the change is successful it should help unblock gitlab-org users which currently encounter disfunction at issues level on boards.

Change Details

  1. Services Impacted - ServicePatroni
  2. Change Technician - @acroitor
  3. Change Criticality - C2
  4. Change Type - changeunscheduled
  5. Change Reviewer - @ahmadsherif
  6. Due Date - Date and time (in UTC) for the execution of the change
  7. Time tracking - Time, in minutes, needed to execute all change steps, including rollback
  8. Downtime Component - If there is a need for downtime, include downtime estimate here

Detailed steps for the change

Pre-Change Steps - steps to be completed before execution of the change

Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes

  • Rails console access
  • Enable rebalance_issues FF for gitlab-org
  • Enable issue_rebalancing_optimization FF
  • Prepare a modified version of the IssueRebalancingService to allow for a bigger number of issues
class IssueRebalancingService
  MAX_ISSUE_COUNT = 200_000
end 
...

Change Steps - steps to take to execute the change

Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes

  • Run IssueRebalancingService - ~3 mins. 
    IssueRebalancingService.new(Project.find(278964).issues.take).execute
  • on success of the step 1 also run IssuePlacementWorker to place accumulated issues with null relative position to the end of the lists.

Post-Change Steps - steps to take to verify the change

Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes

  • Check there are no more issues with relative position = NULL on gitlab-org
  • Check the min and max relative positions on gitlab-org
  • Disable rebalance_issues FF for gitlab-org
  • Disable issue_rebalancing_optimization FF

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes

  • Auto rollbacks in case of a query timeout as it is being run in a transaction.
  • Disable rebalance_issues FF for gitlab-org
  • Disable issue_rebalancing_optimization FF

Monitoring

Key metrics to observe

  • Metric: Metric Name
    • Location: Dashboard URL
    • What changes to this metric should prompt a rollback: Describe Changes

Summary of infrastructure changes

  • Does this change introduce new compute instances?
  • Does this change re-size any existing compute instances?
  • Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?

Summary of the above

Changes checklist

  • This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities.
  • This issue has the change technician as the assignee.
  • Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed.
  • Necessary approvals have been completed based on the Change Management Workflow.
  • Change has been tested in staging and results noted in a comment on this issue.
  • A dry-run has been conducted and results noted in a comment on this issue.
  • SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall and this issue and await their acknowledgement.)
  • There are currently no active incidents.
Edited by Nels Nelson