Skip to content

Database snapshots should support GITLAB_BACKUP_* overrides

Rutger Wessels requested to merge 405159-db-backup-snapshot-fix into master

What does this MR do and why?

In !107726 (merged), we introduced using PostgreSQL snapshots for taking consistent backups of multiple database. This is working fine in case the backups are using configuration from database.yml.

When using tooling like PgBouncer, we advise administrators to override database.yml by using environment variables. These enviroment variables are used to instruct pg_dump to connect to another database server.

But since snapshots are taken using ActiveRecord connections, we also need to override ActiveRecord in case GITLAB_BACKUP_* overrides are present, but only for database backup purposes

Changes in this MR:

  • Allow overriding ActiveRecord configuration so snapshots are using the same database credentials as the pg_dump executable
  • The ENV['PG_*'] variables are now also unset.

Related to #405159 (closed)

Screenshots or screen recordings

How to set up and validate locally

using GDK, tested backup and restore for:

  • ✅ single database setup
  • ✅ single database, two connections
  • ✅ multiple databases

Also tested with overriding database.yml:

GITLAB_BACKUP_PGHOST=localhost GITLAB_BACKUP_PGPORT=15432 SKIP=tar,uploads,repositories,builds,artifacts,lfs,registry,pages,terraform_state bash test.sh
# Test backups

echo "=> Deleting existing backups in tmp/backups..."
rm -Rf tmp/backups

echo "=> Both tables should have some data that will be part of the backup"
gdk psql -c "INSERT INTO ci_runner_versions values ('Database ci: this row will be part of the backup', 1);" -d gitlabhq_development_ci
gdk psql -c "SELECT * FROM shards;"
gdk psql -c "SELECT * FROM ci_runner_versions;" -d gitlabhq_development_ci

echo "=> Running database backup script"
SKIP=uploads,repositories,builds,artifacts,lfs,registry,pages,terraform_state bundle exec rake gitlab:backup:create

echo "=> Insert some data"
gdk psql -c "INSERT INTO shards (name) VALUES ('Database main: a restore will remove this');"
gdk psql -c "SELECT * FROM shards;"
gdk psql -c "INSERT INTO ci_runner_versions values ('Database ci: a restore will remove this', 1);" -d gitlabhq_development_ci
gdk psql -c "SELECT * FROM ci_runner_versions;" -d gitlabhq_development_ci

echo "=> So we now have data in both tables. Starting restore, which should"
echo "=> not restore the data that was added after backup was created"
sleep 5

bundle exec rake gitlab:backup:restore

echo "=> Restore done, let's check inserted data, created data should be gone"
gdk psql -c "SELECT * FROM shards;"
gdk psql -c "SELECT * FROM ci_runner_versions;" -d gitlabhq_development_ci
sleep 5

gdk psql -c "DELETE FROM shards WHERE name LIKE 'Database main: a restore will remove this';"
gdk psql -c "DELETE FROM ci_runner_versions WHERE version LIKE 'Database ci%';" -d gitlabhq_development_ci

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Rutger Wessels

Merge request reports