Move version.gitlab.com to GKE and Auto Devops
C2
Production Change - Criticality 2Change Objective | Moving version.gitlab.com to Google Kubernetes Engine and enabling Auto Devops |
---|---|
Change Type | Type described above |
Services Impacted | version.gitlab.com |
Change Team Members | @devin |
Change Severity | C2 |
Buddy check | A colleague will review the change |
Tested in staging | The application has been tested in the new staging environment |
Schedule of the change | Wednesday, October 30th. 13:00HST (23:00UTC) |
Duration of the change | TBD (current estimate 4 hours) |
Detailed steps for the change. Each step must include: | - pre-conditions for execution of the step, - execution commands for the step, - post-execution validation for the step , - rollback of the step |
Steps:
Shut down old production instance
- Stop chef on the old
version.gitlab.com
machine - Stop nginx and unicorn
- Leave Postgres running
Dump data:
After verifying agent forwarding, the following will be run on the temporary GCP migration instance:
ssh devin@version.gitlab.com 'sudo sudo -u postgres pg_dump -c version_gitlab_com_production | gzip' > data.gz
cat data.gz | gunzip | gcloud sql connect cloudsql-5909 --user default
We will do it in two steps like that because in testing, there were two many failures when doing it as a single pipeline like this:
ssh devin@version.gitlab.com 'sudo sudo -u postgres pg_dump -c version_gitlab_com_production | gzip' | gunzip | gcloud sql connect cloudsql-4cb7 --user default
When restoring the data into the new instance. the temp_file
flag must be set to a much higher value
Test new production instance
According to this comment, it should be sufficient to simply visit each tab in the header of the application to validate that everything is working properly.
Switch DNS
Connect to AWS Route53 console, and update the IP of the version.gitlab.com
entry to point to the ingress IP for the production cluster. After this has time to propagate, it will be necessary to push out another deployment in order to force the Lets Encrypt certificate agent to request a new certificate for the newly changed DNS name.
Cleanup
The old instance uses https://gitlab.com/gitlab-cookbooks/cookbook-version-gitlab-com for configuration. Once the move is complete and we are confident that we won't have to roll back, this chef code can be deprecated.
The old virtual machine is in AWS. It should be archived and removed.
Production Readiness Review readiness!11 (merged)