Switch to Sentry on GCP

Background

Part of migrating Sentry to GCP is to upgrade the DB. Since we can't use WAL-E replication between two major versions, we have a 9.3 DB doing the replication (currently up-to-date, give or take couple of WAL segments delay), then we upgrade it to 9.5.

To avoid data loss, we need to shutdown Azure Sentry for at least 35 minutes (the upgrade duration).

Steps

  • Shutdown Azure Sentry
    • sudo systemctl stop supervisor
  • Create a new database object on Azure Sentry to mark the end of data
    • sudo -u postgres psql -c 'create database tombstone;'
  • Run CHECKPOINT to force a checkpoint
    • sudo -u postgres psql -c 'checkpoint;'
  • Wait a few minutes for GCP Sentry to consume the latest WAL segments, wait until the tombstone can be seen
  • Shutdown Postgres in Sentry Azure
    • sudo systemctl stop postgresql
  • In GCP Sentry, move Postgres 9.5 data to a backup location
    • mv /var/lib/postgresql/9.5{,.bak}
    • mv /etc/postgresql/9.5{,.bak}
  • Make Postgres 9.3 a master
    • mv /var/lib/postgresql/9.3/main/recovery.{conf,done} && sudo systemctl restart postgresql@9.3-main.service
  • Run the upgrade command
    • sudo pg_upgradecluster -m upgrade -v 9.5 9.3 main
    • It takes around 35 minutes to complete

Successful Upgrade

  • Change the port for Postgres 9.5 to 5432 as the upgrade script switches to the port used by the old cluster
    • vim /etc/postgresql/9.5/main/postgresql.conf
  • Start Postgresql 9.5 then Sentry
    • sudo systemctl start postgresql@9.5-main.service && sudo systemctl start supervisor
  • Make sure Sentry is usable at https://sentry.gitlab.net
  • Make sure Sentry is receiving exceptions on staging
    • In chef-repo: git checkout switch-to-new-sentry && knife role from file roles/gstg-base.json && knife ssh roles:gstg-base-fe-web 'sudo chef-client'
    • In a Rails console on web-01-sv-gstg.c.gitlab-staging-1.internal, run app.get 'https://staging.gitlab.com/ahmadsherif-sandbox/throw-500/'
    • Make sure there's a matching exception on Sentry side in the staginggitlabcom project
  • Update sentry.gitlap.net DNS record to be a CNAME with the value sentry.gitlab.net.
  • Monitor the other Sentry projects and see if they are receiving exceptions
    • If not, move to the next step then HUP/restart services gradually, keep monitoring
  • Merge and apply https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/2455

Upgrade Failure

  • Restart Azure Sentry
    • sudo systemctl stop postgresql && sudo systemctl stop supervisor

cc/ @gl-infra @glopezfernandez @andrewn

Edited by Ahmad Sherif