Add rake task for copying 'main' database to 'ci' database

What does this MR do and why?

We are working towards making decomposed database setup a Beta feature. As part of that, it should be more easy for self-managed customers to migrate from one-database setup to decomposed two-database setup.

This MR adds a rake task that will take care of dumping the gitlabhq_production database and importing it in gitlabhq_production_ci database. This rake task will be used by all installation methods that we support.

This rake tasks will be called by installation-specific scripts. A first follow-up MR will be to add the script to Omnibus package. We can then iterate on this to have scripts for other installation methods.

There are some checks before we do anything:

• Check if GitLab is already decomposed
• Ensure we have enough local disk space for the dump of main database
• Ensure the new gitlabhq_production_ci is accessible and empty
• Ensure there are no active running Background Migrations

The dump is sharing some code with the Backup tasks. This makes it possible to override default database settings using the same environment variables as we support for Backups. Because using GITLAB_BACKUP_PGHOST variables names looked weird, I added support for using GITLAB_OVERRIDE_PGHOST

The database dump is using pg_dump -Fd, this creates dump files for each table. This allows us to dump and restore using multiple processes.

Related to #368729 (closed)

How to set up and validate locally

Using GDK:

1. Modify database.yml: comment out ci section (so Rails is using a single database)
2. Drop and create ci database gdk psql -c "DROP DATABASE gitlabhq_development_ci";gdk psql -c "CREATE DATABASE gitlabhq_development_ci"
3. Run the rake task: bundle exec rake gitlab:db:decomposition:migrate
4. Verify CI database now also have projects: gdk psql -d gitlabhq_development -c "SELECT COUNT(*) FROM projects";gdk psql -d gitlabhq_development_ci -c "SELECT COUNT(*) FROM projects"; should match

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

changed milestone to %16.6

added Category:Cell backend customer devopsdata stores grouptenant scale milestonep1 release post item release post itemin review sectioncore platform self-managed-decompositionbeta typefeature workflowready for development labels

assigned to @rutgerwessels

added 1 commit

• 5e1c6dc2 - Add class for switching to Decomposed DB setup

Compare with previous version

Allure report

allure-report-publisher generated test report!

e2e-test-on-gdk: test report for 2a808ac2

expand test summary

+-----------------------------------------------------------------------+
|                            suites summary                             |
+------------------+--------+--------+---------+-------+-------+--------+
|                  | passed | failed | skipped | flaky | total | result |
+------------------+--------+--------+---------+-------+-------+--------+
| Plan             | 55     | 0      | 0       | 0     | 55    | ✅     |
| Data Stores      | 23     | 0      | 0       | 0     | 23    | ✅     |
| Verify           | 31     | 0      | 0       | 0     | 31    | ✅     |
| Govern           | 57     | 0      | 0       | 0     | 57    | ✅     |
| Create           | 48     | 0      | 9       | 0     | 57    | ✅     |
| Package          | 0      | 0      | 1       | 0     | 1     | ➖     |
| Monitor          | 8      | 0      | 0       | 0     | 8     | ✅     |
| Framework sanity | 0      | 0      | 1       | 0     | 1     | ➖     |
| Manage           | 0      | 0      | 1       | 0     | 1     | ➖     |
+------------------+--------+--------+---------+-------+-------+--------+
| Total            | 222    | 0      | 12      | 0     | 234   | ✅     |
+------------------+--------+--------+---------+-------+-------+--------+

added 1367 commits

• 5e1c6dc2...0ac8e8e1 - 1359 commits from branch master
• 039c44ff - Add class for switching to Decomposed DB setup
• d21d1727 - Add check for available local diskspace
• 63d30fb1 - Add check for single db setup
• ffe1e10a - Add check for ensuring ci database has been created
• 036e55d3 - Add check for ensuring ci database is empty
• 15f67e4b - More generic message if ci database is not found
• 352fdd9c - Add rake task for migrating single DB to two db setup
• 5dd7d052 - Allow overrriding backup location

Compare with previous version

marked this merge request as ready

changed title from Draft: Create a tool to migrate existing self-managed customers to decomposed to Add rake task to copy 'main' database to 'ci' database

changed the description

added database databasereview pending workflowin review labels and removed workflowready for development label

removed release post item label

removed customer label

added 1 commit

• 04acf705 - Randomize temporary backup location name

Compare with previous version

added 568 commits

• 04acf705...e93158c2 - 558 commits from branch master
• 1a8ac9dd - Add class for switching to Decomposed DB setup
• a12cd5cf - Add check for available local diskspace
• 7ad4602c - Add check for single db setup
• 021bba9c - Add check for ensuring ci database has been created
• 2222734f - Add check for ensuring ci database is empty
• 57d809d1 - More generic message if ci database is not found
• 99200f8b - Add rake task for migrating single DB to two db setup
• 3b2b32c9 - Allow overrriding backup location
• 2ac188c1 - Randomize temporary backup location name
• 5359c792 - Add bash script for migrating to decomposed setup

Compare with previous version

	1 Warning
	6c7c3749: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines.

	1 Message
	CHANGELOG missing: If this merge request needs a changelog entry, add the Changelog trailer to the commit message you want to add to the changelog. If this merge request doesn't need a CHANGELOG entry, feel free to ignore this message.

Reviewer roulette

Changes that require review have been detected!

Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:

Category	Reviewer	Maintainer
backend	@morefice (UTC+1, same timezone as author)	@tkuah (UTC+13, 12 hours ahead of author)
database	@johnmason (UTC-5, 6 hours behind author)	@ahegyi (UTC+1, same timezone as author)

Please check reviewer's status!

• Reviewer is available!
• Reviewer is unavailable!

Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.

To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.

Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.

If needed, you can retry the danger-review job that generated this comment.

Generated by Danger

changed title from Add rake task to copy 'main' database to 'ci' database to Omnibus: Add migration script for decomposing GitLab self-managed installations

changed the description

marked this merge request as draft