GitLab Next

Production Change - Criticality 2 C2

Related to infrastructure#7356 (closed)

Change Objective	Describe the objective of the change
Change Type	ConfigurationChange
Services Impacted	ServiceWeb ServiceSidekiq ServiceAPI ServiceGit
Change Team Members	@ahmadsherif
Change Criticality	C2
Change Reviewer	A colleague who will review the change
Tested in staging	It has been running in staging for a quite a while infrastructure#8184 (closed)
Dry-run output
Due Date	2020-03-27
Time tracking	1 hour

Detailed steps for the change

1. Stop Chef on the front-end nodes that are running Puma
   • knife ssh 'roles:gprd-base-fe AND omnibus-gitlab_gitlab_rb_puma_enable:true' 'sudo systemctl stop chef-client'
2. Stop Chef on the Sidekiq nodes
   • knife ssh 'roles:gprd-base-be-sidekiq' 'sudo systemctl stop chef-client'
3. Merge and apply https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/2982
4. Node by node (4 at a time in parallel for efficiency), drain from haproxy, run/enable chef, and undrain:

   ./bin/set-server-state gprd drain NODE
   ssh NODE "sudo chef-client"
   ./bin/set-server-state gprd ready NODE

5. Converge the Sidekiq nodes
   • knife ssh -C3 'roles:gprd-base-be-sidekiq' 'sudo chef-client'

Monitor

1. Sentry for any diff-related errors https://sentry.gitlab.net/gitlab/gitlabcom/
2. Triage dashboard for an overview of any blips https://dashboards.gitlab.net/d/RZmbBr7mk/gitlab-triage?orgId=1&refresh=30s

Rollback steps

1. Revert and apply https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/2982
2. Converge Chef

Migrate the files back to the database

1. Compile a list of migrated diff files
   • There's no index that can be used to find the migrated records, so the following query should be run in non-production database (e.g. archive-replica).
   • MergeRequestDiff.where(stored_externally: true).pluck(:id)
2. Transfer the list of IDs to a console node, run something like:
   • ids_list.with_groups_of(100) { |ids| MergeRequestDiff.find(ids).each(&:migrate_files_to_database!) }

Changes checklist

• Detailed steps and rollback steps have been filled prior to commencing work
• Person on-call has been informed prior to change being rolled out