Migrate Packagecloud

Production Change - Criticality 3 C3

Change Objective	Describe the objective of the change
Change Type	Operation
Services Impacted	Packagecloud
Change Team Members	@cindy
Change Severity	C3
Change Reviewer or tested in staging	A colleague who will review the change or evidence the change was tested on staging environment
Dry-run output	If the change is done through a script, it is mandatory to have a dry-run capability in the script, run the change in dry-run mode and output the result
Due Date	2020-03-02T22:00:00
Time tracking	3 hours

Detailed steps for the change

Pre-steps

• Mount EBS volume to /var/opt/packagecloud
• Install chef-client on new node
• Configure IPv6
• Bootstrap new node
  • Configure run lists and secrets
• Install packagecloud (v2.0.9) on new node
• Make MR to switch UUID in the chef-repo

Steps to migration

• Stop chef client on packagecloud instance with sudo service chef-client stop
• Stop nginx with packagecloud-ctl stop nginx
• Merge MR to change UUID in chef-repo and apply to new server
• Take a backup from original packagecloud instance and upload to S3 (will take about 1.5 hours) with sudo packagecloud-ctl backup-all
• Restore backup to new node (~45-50 minutes)
  • Download the most recent backup from s3://gitlab-packagecloud-db-backups/backups and place it in /var/opt/packagecloud/backups/. Used scp due to time.
  • Run packagecloud-ctl backup-database-restore /var/opt/packagecloud/backups/packagecloud-database-backup.tgz
• Upgrade packagecloud to v3.0.0 with sudo apt install packagecloud=3.0.0-1
• Test fetching a GitLab package
• Attach elastic IP to new node

Rollback steps

• Start nginx with packagecloud-ctl start nginx
• Re-attach elastic IP to old packagecloud instance
• Rename chef node from old-packages.gitlab.com to packages.gitlab.com

Changes checklist

• Detailed steps and rollback steps have been filled prior to commencing work
• Person on-call has been informed prior to change being rolled out