Add rake task for copying 'main' database to 'ci' database
What does this MR do and why?
We are working towards making decomposed database setup a Beta feature. As part of that, it should be more easy for self-managed customers to migrate from one-database setup to decomposed two-database setup.
This MR adds a rake task that will take care of dumping the gitlabhq_production
database and importing it in gitlabhq_production_ci
database. This rake task will be used by all installation methods that we support.
This rake tasks will be called by installation-specific scripts. A first follow-up MR will be to add the script to Omnibus package. We can then iterate on this to have scripts for other installation methods.
There are some checks before we do anything:
- Check if GitLab is already decomposed
- Ensure we have enough local disk space for the dump of
main
database - Ensure the new
gitlabhq_production_ci
is accessible and empty - Ensure there are no active running Background Migrations
The dump is sharing some code with the Backup tasks. This makes it possible to override default database settings using the same environment variables as we support for Backups. Because using GITLAB_BACKUP_PGHOST
variables names looked weird, I added support for using GITLAB_OVERRIDE_PGHOST
The database dump is using pg_dump -Fd
, this creates dump files for each table. This allows us to dump and restore using multiple processes.
Related to #368729 (closed)
How to set up and validate locally
Using GDK:
- Modify
database.yml
: comment outci
section (so Rails is using a single database) - Drop and create ci database
gdk psql -c "DROP DATABASE gitlabhq_development_ci";gdk psql -c "CREATE DATABASE gitlabhq_development_ci"
- Run the rake task:
bundle exec rake gitlab:db:decomposition:migrate
- Verify CI database now also have projects:
gdk psql -d gitlabhq_development -c "SELECT COUNT(*) FROM projects";gdk psql -d gitlabhq_development_ci -c "SELECT COUNT(*) FROM projects";
should match
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Merge request reports
Activity
changed milestone to %16.6
assigned to @rutgerwessels
added 1 commit
- 5e1c6dc2 - Add class for switching to Decomposed DB setup
Allure report
allure-report-publisher
generated test report!e2e-test-on-gdk:
test report for 2a808ac2expand test summary
+-----------------------------------------------------------------------+ | suites summary | +------------------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +------------------+--------+--------+---------+-------+-------+--------+ | Plan | 55 | 0 | 0 | 0 | 55 | ✅ | | Data Stores | 23 | 0 | 0 | 0 | 23 | ✅ | | Verify | 31 | 0 | 0 | 0 | 31 | ✅ | | Govern | 57 | 0 | 0 | 0 | 57 | ✅ | | Create | 48 | 0 | 9 | 0 | 57 | ✅ | | Package | 0 | 0 | 1 | 0 | 1 | ➖ | | Monitor | 8 | 0 | 0 | 0 | 8 | ✅ | | Framework sanity | 0 | 0 | 1 | 0 | 1 | ➖ | | Manage | 0 | 0 | 1 | 0 | 1 | ➖ | +------------------+--------+--------+---------+-------+-------+--------+ | Total | 222 | 0 | 12 | 0 | 234 | ✅ | +------------------+--------+--------+---------+-------+-------+--------+
added 1367 commits
-
5e1c6dc2...0ac8e8e1 - 1359 commits from branch
master
- 039c44ff - Add class for switching to Decomposed DB setup
- d21d1727 - Add check for available local diskspace
- 63d30fb1 - Add check for single db setup
- ffe1e10a - Add check for ensuring ci database has been created
- 036e55d3 - Add check for ensuring ci database is empty
- 15f67e4b - More generic message if ci database is not found
- 352fdd9c - Add rake task for migrating single DB to two db setup
- 5dd7d052 - Allow overrriding backup location
Toggle commit list-
5e1c6dc2...0ac8e8e1 - 1359 commits from branch
added database databasereview pending workflowin review labels and removed workflowready for development label
removed release post item label
removed customer label
added 568 commits
-
04acf705...e93158c2 - 558 commits from branch
master
- 1a8ac9dd - Add class for switching to Decomposed DB setup
- a12cd5cf - Add check for available local diskspace
- 7ad4602c - Add check for single db setup
- 021bba9c - Add check for ensuring ci database has been created
- 2222734f - Add check for ensuring ci database is empty
- 57d809d1 - More generic message if ci database is not found
- 99200f8b - Add rake task for migrating single DB to two db setup
- 3b2b32c9 - Allow overrriding backup location
- 2ac188c1 - Randomize temporary backup location name
- 5359c792 - Add bash script for migrating to decomposed setup
Toggle commit list-
04acf705...e93158c2 - 558 commits from branch
1 Warning 6c7c3749: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines. 1 Message CHANGELOG missing: If this merge request needs a changelog entry, add the
Changelog
trailer to the commit message you want to add to the changelog.If this merge request doesn't need a CHANGELOG entry, feel free to ignore this message.
Reviewer roulette
Changes that require review have been detected!
Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:
Category Reviewer Maintainer backend @morefice
(UTC+1, same timezone as author)
@tkuah
(UTC+13, 12 hours ahead of author)
database @johnmason
(UTC-5, 6 hours behind author)
@ahegyi
(UTC+1, same timezone as author)
Please check reviewer's status!
Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.
To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.
Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.
If needed, you can retry the
danger-review
job that generated this comment.Generated by
Dangeradded 89 commits
-
5359c792...3a401474 - 78 commits from branch
master
- 85a41cac - 1 earlier commit
- c57460be - Add check for available local diskspace
- 4ba2d0db - Add check for single db setup
- 07bfdf51 - Add check for ensuring ci database has been created
- c9c635bb - Add check for ensuring ci database is empty
- 1bf943c4 - More generic message if ci database is not found
- cfe29f27 - Add rake task for migrating single DB to two db setup
- f89b5bd1 - Allow overrriding backup location
- 883d4428 - Randomize temporary backup location name
- fe6389c4 - Add bash script for migrating to decomposed setup
- 024e375f - Support GITLAB_OVERRIDE_* environment variables
Toggle commit list-
5359c792...3a401474 - 78 commits from branch
requested review from @tkuah
- Resolved by Rutger Wessels
Hi @tkuah Can you please review this, also from functional point of view? Are there any features missing? I try to make this foolproof, I hope I covered the most common edge cases.
added 5 commits
Toggle commit listadded 224 commits
-
28b537ef...9ffecb10 - 213 commits from branch
master
- 7043dd07 - 1 earlier commit
- 9294a670 - Add check for available local diskspace
- 483eca62 - Add check for single db setup
- a84f6c8c - Add check for ensuring ci database has been created
- fe29966e - Add check for ensuring ci database is empty
- 2d3243bd - More generic message if ci database is not found
- bf230a8a - Add rake task for migrating single DB to two db setup
- 98e44a15 - Allow overrriding backup location
- 95dbd210 - Randomize temporary backup location name
- 46363516 - Add bash script for migrating to decomposed setup
- 09ea3765 - Support GITLAB_OVERRIDE_* environment variables
Toggle commit list-
28b537ef...9ffecb10 - 213 commits from branch
changed milestone to %16.7
added missed:16.6 label
- Resolved by Thong Kuah
- Resolved by Rutger Wessels
- Resolved by Rutger Wessels
- Resolved by Thong Kuah
- Resolved by Rutger Wessels
- Resolved by Thong Kuah
- Resolved by Rutger Wessels
- Resolved by Rutger Wessels
- Resolved by Rutger Wessels
- Resolved by Thong Kuah
removed review request for @tkuah
- Resolved by Thong Kuah
@tkuah Based on your comments, I start to wonder if we should split this migration script into two parts: a
migration
part and apost-migration
part. This allows us to let the administrator do some manual work. The process could then be:- Admin will run
migrate
task (which will do the pre-flight checks and copy the data) - Pause Admin will:
- Manually review the result of the copy (ie do the database sizes match?)
- Edit the
/etc/gitlab/gitlab.rb
file (we can display the changes instead of updating the file by a script)
- Admin run
post-migrate
bash script:- gitlab-ctl reconfigure
- gitlab-rake gitlab:db:lock_writes
- gitlab-rails runner "Feature.enable(:execute_background_migrations) && Feature.enable(:execute_batched_migrations_on_schedule)"
- Admin can then decide to restart gitlab
What do you think? Administrators do have then a 'mid-migration' moment that allows them to verify the database copy and, in case of problems, allows them to cancel the migration process and restart GitLab
- Admin will run
added 348 commits
-
09ea3765...35254336 - 336 commits from branch
master
- 35254336...fb8483f5 - 2 earlier commits
- 77d967f5 - Add check for single db setup
- 2499c84c - Add check for ensuring ci database has been created
- 1ba30512 - Add check for ensuring ci database is empty
- a9a2faf4 - More generic message if ci database is not found
- cf6f7905 - Add rake task for migrating single DB to two db setup
- 5e036c89 - Allow overrriding backup location
- 97bd27f6 - Randomize temporary backup location name
- 285008bf - Add bash script for migrating to decomposed setup
- df573c2b - Support GITLAB_OVERRIDE_* environment variables
- 7678d76b - Allow configuring backup base location
Toggle commit list-
09ea3765...35254336 - 336 commits from branch
added 484 commits
-
7678d76b...ae3b4d35 - 469 commits from branch
master
- ae3b4d35...f2455e71 - 5 earlier commits
- 8197f686 - More generic message if ci database is not found
- 3d7221ae - Add rake task for migrating single DB to two db setup
- 8348a959 - Allow overrriding backup location
- 35a2cab1 - Randomize temporary backup location name
- 2b54a982 - Add bash script for migrating to decomposed setup
- 982dc19a - Support GITLAB_OVERRIDE_* environment variables
- 6c1e9a19 - Allow configuring backup base location
- fc92e0fd - Allow administrator to edit config
- cf0153a4 - Moved background migration check out of shell script
- c3bf31be - Remove bash scripts
Toggle commit list-
7678d76b...ae3b4d35 - 469 commits from branch
added 484 commits
-
7678d76b...ae3b4d35 - 469 commits from branch
master
- ae3b4d35...f2455e71 - 5 earlier commits
- 8197f686 - More generic message if ci database is not found
- 3d7221ae - Add rake task for migrating single DB to two db setup
- 8348a959 - Allow overrriding backup location
- 35a2cab1 - Randomize temporary backup location name
- 2b54a982 - Add bash script for migrating to decomposed setup
- 982dc19a - Support GITLAB_OVERRIDE_* environment variables
- 6c1e9a19 - Allow configuring backup base location
- fc92e0fd - Allow administrator to edit config
- cf0153a4 - Moved background migration check out of shell script
- c3bf31be - Remove bash scripts
Toggle commit list-
7678d76b...ae3b4d35 - 469 commits from branch
- Resolved by Michał Zając
- Resolved by Rutger Wessels
- Resolved by Thong Kuah
@tkuah
, thanks for approving this merge request.This is the first time the merge request has been approved. To ensure we don't only run predictive pipelines, and we don't break
master
, a new pipeline will be started shortly.Please wait for the pipeline to start before resolving this discussion and set auto-merge for the new pipeline. See merging a merge request for more details.
added pipeline:mr-approved label
- Resolved by Rutger Wessels
LGTM @rutgerwessels !
@jarka Can you please review as maintainer ?
added 146 commits
-
c3bf31be...943a51dc - 130 commits from branch
master
- 943a51dc...64110428 - 6 earlier commits
- 1f3c5680 - Add rake task for migrating single DB to two db setup
- e3fba61a - Allow overrriding backup location
- a8777e18 - Randomize temporary backup location name
- 52a0ca22 - Add bash script for migrating to decomposed setup
- 604f0375 - Support GITLAB_OVERRIDE_* environment variables
- 1c8b5940 - Allow configuring backup base location
- e929bd1f - Allow administrator to edit config
- ce87e259 - Moved background migration check out of shell script
- d5c4079c - Remove bash scripts
- e51376fd - Remove typo
Toggle commit list-
c3bf31be...943a51dc - 130 commits from branch
- Resolved by Rutger Wessels
removed review request for @jarka
added pipeline:skip-undercoverage label
requested review from @jarka
- Resolved by Terri Chu
- Resolved by Rutger Wessels
- Resolved by Rutger Wessels
- Resolved by Terri Chu
- Resolved by Terri Chu
removed review request for @terrichu
- Resolved by Terri Chu
added 80 commits
-
e51376fd...b42e77e3 - 62 commits from branch
master
- b42e77e3...bb3953ea - 8 earlier commits
- fe025bd5 - Randomize temporary backup location name
- 9d9c0702 - Add bash script for migrating to decomposed setup
- b579a246 - Support GITLAB_OVERRIDE_* environment variables
- 099305d3 - Allow configuring backup base location
- c0f943cc - Allow administrator to edit config
- 79d7edca - Moved background migration check out of shell script
- 94c9876a - Remove bash scripts
- 2f2a73a2 - Remove typo
- a866284b - Only create test data for the test that actually duplicates the data
- a92e4ebb - Extract diskpace headroom factor into a constant
Toggle commit list-
e51376fd...b42e77e3 - 62 commits from branch
requested review from @terrichu
added databasereviewed label and removed databasereview pending label
requested review from @Quintasan and removed review request for @terrichu
removed review request for @jarka
added 623 commits
-
a92e4ebb...1e639b8f - 604 commits from branch
master
- 1e639b8f...8a132034 - 9 earlier commits
- 0d0cc293 - Add bash script for migrating to decomposed setup
- 92b84a3c - Support GITLAB_OVERRIDE_* environment variables
- cad4c7a1 - Allow configuring backup base location
- 4dfdf901 - Allow administrator to edit config
- db9d89ad - Moved background migration check out of shell script
- b8c7ac00 - Remove bash scripts
- f9631f1d - Remove typo
- f0e8679e - Only create test data for the test that actually duplicates the data
- a2487dcc - Extract diskpace headroom factor into a constant
- e546fc2a - Create directory if it does not exist
Toggle commit list-
a92e4ebb...1e639b8f - 604 commits from branch
added 614 commits
-
e546fc2a...c916a646 - 595 commits from branch
master
- c916a646...2157b194 - 9 earlier commits
- d3e92d1b - Add bash script for migrating to decomposed setup
- 3079892b - Support GITLAB_OVERRIDE_* environment variables
- 6706ff64 - Allow configuring backup base location
- 93df3d89 - Allow administrator to edit config
- bacb7b01 - Moved background migration check out of shell script
- fa79daab - Remove bash scripts
- 6c7c3749 - Remove typo
- c5815a30 - Only create test data for the test that actually duplicates the data
- e012d506 - Extract diskpace headroom factor into a constant
- 2a808ac2 - Create directory if it does not exist
Toggle commit list-
e546fc2a...c916a646 - 595 commits from branch
@Quintasan Can you please have a look? I did start a new CI build because there was one failing job (danger-local) but no spec failures.
added databaseapproved label and removed databasereviewed label
enabled an automatic merge when the pipeline for dff6cac9 succeeds
Hello @rutgerwessels
The database team is looking for ways to improve the database review process and we would love your help!
If you'd be open to someone on the database team reaching out to you for a chat, or if you'd like to leave some feedback asynchronously, just post a reply to this comment mentioning:
@gitlab-org/database-team
And someone will be by shortly!
Thanks for your help!
This message was generated automatically. You're welcome to improve it.
mentioned in commit 743cf835
added workflowstaging-canary label and removed workflowin review label
added workflowcanary label and removed workflowstaging-canary label
added workflowstaging label and removed workflowcanary label
added workflowproduction label and removed workflowstaging label
mentioned in issue #368729 (closed)
mentioned in merge request omnibus-gitlab!7266 (merged)
added workflowpost-deploy-db-staging label and removed workflowproduction label
added workflowpost-deploy-db-production label and removed workflowpost-deploy-db-staging label
added releasedcandidate label
mentioned in merge request !140122 (merged)
- lib/gitlab/database/decomposition/migrate.rb 0 → 100644
110 output, status = with_transient_pg_env(ci_config[:pg_env]) do 111 psql_args = ["--dbname=#{ci_database_name}", "-tAc", sql] 112 113 Open3.capture2e('psql', *psql_args) 114 end 115 116 unless status.success? && output.chomp.to_i == 0 117 raise MigrateError, 118 "Database '#{ci_database_name}' is not empty" 119 end 120 121 true 122 end 123 124 def background_migrations_done? 125 unfinished_count = Gitlab::Database::BackgroundMigration::BatchedMigration.without_status(:finished).count @rutgerwessels / @Quintasan - what about status finalized (6) here - cannot run migration in omnibus because 19 of my background migrations are in status finalized (6). (trying in GitLab 17.0 currently).
for now i have managed to migrate successfully by executing (ids identified by filtering migrations with status 6).
pre-decomposition in gitlabhq_production:
update batched_background_migrations set status = 3 where id in (211, 210, 209, 208, 206, 203, 202, 195, 193, 186, 182, 181, 180, 179, 164, 106, 103, 41, 38)
post-decomposition in gitlabhq_production & gitlabhq_production_ci:
update batched_background_migrations set status = 6 where id in (211, 210, 209, 208, 206, 203, 202, 195, 193, 186, 182, 181, 180, 179, 164, 106, 103, 41, 38)
@swiffer Thanks, good find. The intention is that we don't have running background migrations during the migration. I will create an issue for this.
Edited by Rutger Wessels
added pipelinetier-3 label
mentioned in issue #462424
added devopstenant scale groupcells infrastructure sectioninfrastructure platforms labels and removed devopsdata stores grouptenant scale [DEPRECATED] sectioncore platform labels
mentioned in issue #474638 (closed)