Skip to content

Add rake task for copying 'main' database to 'ci' database

Rutger Wessels requested to merge 368729-create-a-tool-to-migrate-existing-self-managed-customers-to-decomposed into master

What does this MR do and why?

We are working towards making decomposed database setup a Beta feature. As part of that, it should be more easy for self-managed customers to migrate from one-database setup to decomposed two-database setup.

This MR adds a rake task that will take care of dumping the gitlabhq_production database and importing it in gitlabhq_production_ci database. This rake task will be used by all installation methods that we support.

This rake tasks will be called by installation-specific scripts. A first follow-up MR will be to add the script to Omnibus package. We can then iterate on this to have scripts for other installation methods.

There are some checks before we do anything:

  • Check if GitLab is already decomposed
  • Ensure we have enough local disk space for the dump of main database
  • Ensure the new gitlabhq_production_ci is accessible and empty
  • Ensure there are no active running Background Migrations

The dump is sharing some code with the Backup tasks. This makes it possible to override default database settings using the same environment variables as we support for Backups. Because using GITLAB_BACKUP_PGHOST variables names looked weird, I added support for using GITLAB_OVERRIDE_PGHOST

The database dump is using pg_dump -Fd, this creates dump files for each table. This allows us to dump and restore using multiple processes.

Related to #368729 (closed)

How to set up and validate locally

Using GDK:

  1. Modify database.yml: comment out ci section (so Rails is using a single database)
  2. Drop and create ci database gdk psql -c "DROP DATABASE gitlabhq_development_ci";gdk psql -c "CREATE DATABASE gitlabhq_development_ci"
  3. Run the rake task: bundle exec rake gitlab:db:decomposition:migrate
  4. Verify CI database now also have projects: gdk psql -d gitlabhq_development -c "SELECT COUNT(*) FROM projects";gdk psql -d gitlabhq_development_ci -c "SELECT COUNT(*) FROM projects"; should match

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Rutger Wessels

Merge request reports