Migrate Packagecloud
C3
Production Change - Criticality 3Change Objective | Describe the objective of the change |
---|---|
Change Type | Operation |
Services Impacted | Packagecloud |
Change Team Members | @cindy |
Change Severity | C3 |
Change Reviewer or tested in staging | A colleague who will review the change or evidence the change was tested on staging environment |
Dry-run output | If the change is done through a script, it is mandatory to have a dry-run capability in the script, run the change in dry-run mode and output the result |
Due Date | 2020-03-02T22:00:00 |
Time tracking | 3 hours |
Detailed steps for the change
Pre-steps
-
Mount EBS volume to /var/opt/packagecloud
-
Install chef-client on new node -
Configure IPv6 -
Bootstrap new node -
Configure run lists and secrets
-
-
Install packagecloud (v2.0.9) on new node -
Make MR to switch UUID in the chef-repo
Steps to migration
-
Stop chef client on packagecloud instance with sudo service chef-client stop
-
Stop nginx with packagecloud-ctl stop nginx
-
Merge MR to change UUID in chef-repo and apply to new server -
Take a backup from original packagecloud instance and upload to S3 (will take about 1.5 hours) with sudo packagecloud-ctl backup-all
-
Restore backup to new node (~45-50 minutes) -
Download the most recent backup from s3://gitlab-packagecloud-db-backups/backups and place it inUsed/var/opt/packagecloud/backups/
.scp
due to time. -
Run packagecloud-ctl backup-database-restore /var/opt/packagecloud/backups/packagecloud-database-backup.tgz
-
-
Upgrade packagecloud to v3.0.0 with sudo apt install packagecloud=3.0.0-1
-
Test fetching a GitLab package -
Attach elastic IP to new node
Rollback steps
-
Start nginx with packagecloud-ctl start nginx
-
Re-attach elastic IP to old packagecloud instance -
Rename chef node from old-packages.gitlab.com
topackages.gitlab.com
Changes checklist
-
Detailed steps and rollback steps have been filled prior to commencing work -
Person on-call has been informed prior to change being rolled out
Edited by Cindy Pallares 🦉