Enable external merge request diff storage
C2
Production Change - Criticality 2Related to https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/7356
Change Objective | Describe the objective of the change |
---|---|
Change Type | ConfigurationChange |
Services Impacted | ServiceWeb ServiceSidekiq ServiceAPI ServiceGit |
Change Team Members | @ahmadsherif |
Change Criticality | C2 |
Change Reviewer | A colleague who will review the change |
Tested in staging | It has been running in staging for a quite a while https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/8184 |
Dry-run output | |
Due Date | 2020-03-27 |
Time tracking | 1 hour |
Detailed steps for the change
-
Stop Chef on the front-end nodes that are running Puma knife ssh 'roles:gprd-base-fe AND omnibus-gitlab_gitlab_rb_puma_enable:true' 'sudo systemctl stop chef-client'
-
Stop Chef on the Sidekiq nodes knife ssh 'roles:gprd-base-be-sidekiq' 'sudo systemctl stop chef-client'
-
Merge and apply https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/2982 -
Node by node (4 at a time in parallel for efficiency), drain from haproxy, run/enable chef, and undrain: ./bin/set-server-state gprd drain NODE ssh NODE "sudo chef-client" ./bin/set-server-state gprd ready NODE
-
Converge the Sidekiq nodes knife ssh -C3 'roles:gprd-base-be-sidekiq' 'sudo chef-client'
Monitor
- Sentry for any diff-related errors https://sentry.gitlab.net/gitlab/gitlabcom/
- Triage dashboard for an overview of any blips https://dashboards.gitlab.net/d/RZmbBr7mk/gitlab-triage?orgId=1&refresh=30s
Rollback steps
- Revert and apply https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/2982
- Converge Chef
Migrate the files back to the database
- Compile a list of migrated diff files
- There's no index that can be used to find the migrated records, so the following query should be run in non-production database (e.g. archive-replica).
MergeRequestDiff.where(stored_externally: true).pluck(:id)
- Transfer the list of IDs to a console node, run something like:
ids_list.with_groups_of(100) { |ids| MergeRequestDiff.find(ids).each(&:migrate_files_to_database!) }
Changes checklist
-
Detailed steps and rollback steps have been filled prior to commencing work -
Person on-call has been informed prior to change being rolled out
Edited by Ahmad Sherif