Switch to Sentry on GCP
Background
Part of migrating Sentry to GCP is to upgrade the DB. Since we can't use WAL-E replication between two major versions, we have a 9.3 DB doing the replication (currently up-to-date, give or take couple of WAL segments delay), then we upgrade it to 9.5.
To avoid data loss, we need to shutdown Azure Sentry for at least 35 minutes (the upgrade duration).
Steps
- Shutdown Azure Sentry
sudo systemctl stop supervisor
- Create a new database object on Azure Sentry to mark the end of data
sudo -u postgres psql -c 'create database tombstone;'
- Run
CHECKPOINT
to force a checkpointsudo -u postgres psql -c 'checkpoint;'
- Wait a few minutes for GCP Sentry to consume the latest WAL segments, wait until the tombstone can be seen
- Shutdown Postgres in Sentry Azure
sudo systemctl stop postgresql
- In GCP Sentry, move Postgres 9.5 data to a backup location
mv /var/lib/postgresql/9.5{,.bak}
mv /etc/postgresql/9.5{,.bak}
- Make Postgres 9.3 a master
mv /var/lib/postgresql/9.3/main/recovery.{conf,done} && sudo systemctl restart postgresql@9.3-main.service
- Run the upgrade command
sudo pg_upgradecluster -m upgrade -v 9.5 9.3 main
- It takes around 35 minutes to complete
Successful Upgrade
- Change the port for Postgres 9.5 to
5432
as the upgrade script switches to the port used by the old clustervim /etc/postgresql/9.5/main/postgresql.conf
- Start Postgresql 9.5 then Sentry
sudo systemctl start postgresql@9.5-main.service && sudo systemctl start supervisor
- Make sure Sentry is usable at https://sentry.gitlab.net
- Make sure Sentry is receiving exceptions on staging
- In
chef-repo
:git checkout switch-to-new-sentry && knife role from file roles/gstg-base.json && knife ssh roles:gstg-base-fe-web 'sudo chef-client'
- In a Rails console on
web-01-sv-gstg.c.gitlab-staging-1.internal
, runapp.get 'https://staging.gitlab.com/ahmadsherif-sandbox/throw-500/'
- Make sure there's a matching exception on Sentry side in the
staginggitlabcom
project
- In
- Update
sentry.gitlap.net
DNS record to be aCNAME
with the valuesentry.gitlab.net.
- Monitor the other Sentry projects and see if they are receiving exceptions
- If not, move to the next step then HUP/restart services gradually, keep monitoring
- Merge and apply
https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/2455
Upgrade Failure
- Restart Azure Sentry
sudo systemctl stop postgresql && sudo systemctl stop supervisor
cc/ @gl-infra @glopezfernandez @andrewn
Edited by Ahmad Sherif