Add registry-database import command to gitlab-ctl
What does this MR do?
Adds a new import
command under gitlab-ctl registry-database
that wraps the registry database import tool, which provides users with the ability to migrate from filesystem metadata (old registry) to database metadata (new registry).
Syntax
$ gitlab-ctl registry-database import -h
Import filesystem metadata into the database
Usage:
gitlab-ctl registry-database import [options]
Options:
-B, --common-blobs import all blob metadata from common storage
-c, --row-count count and log number of rows across relevant database tables on (pre)import completion
-d, --dry-run do not commit changes to the database
-e, --require-empty-database abort import if the database is not empty
-p, --pre-import import immutable repository-scoped data to speed up a following import
-r, --all-repositories import all repository-scoped data
-h, --help help for import
-1, --step-one pre-import perform step one of a multi-step import: alias for pre-import
-2, --step-two all-repositories perform step two of a multi-step import: alias for all-repositories
-3, --step-three common-blobs perform step three of a multi-step import: alias for common-blobs
For more details about each option please see the documentation linked above.
Validations
- The registry must be enabled before proceeding
- The registry binary exists
- The configuration file exists
- If the registry is running and not set to read-only mode, the execution is aborted when using the
-r/--all-repositories
or-2/--step-two
options
Why?
As of today, users in the Beta program of Release container registry with metadata databa... (&5521) need to bypass gitlab-ctl
and interact directly with the registry CLI to perform a migration. With this change we allow them to keep using gitlab-ctl
, while also providing another layer for basic validations.
Testing
Initial setup
-
Setup a development Omnibus environment
-
Provision a separate PostgreSQL instance and create a logical database, for example:
CREATE USER registry WITH PASSWORD 'registrypassword' CREATE DATABASE registry_omnibus WITH OWNER registry
Alternatively, you can use the provisioned PostgreSQL instance but you will need to allow the registry to connect to it via Unix socket (sample)
-
Edit
/etc/gitlab/gitlab.rb
and enable the registry service:registry['enable'] = true
-
Also configure the database section under the registry configuration (update accordingly):
registry['database'] = { 'enabled' => true, 'host' => '172.17.0.1', 'port' => 5432, 'user' => 'registry', 'password' => 'registrypassword', 'dbname' => 'registry_omnibus', 'sslmode' => 'disable' }
-
Finally, update/add a
storage
section to includemaintenance.readonly.enabled
. Let it set tofalse
for now:registry['storage'] = { 'filesystem' => { 'rootdirectory' => "/var/opt/gitlab/gitlab-rails/shared/registry" }, 'maintenance' => { 'readonly' => { 'enabled' => false } } }
-
To be able to test the actual import we need some sample data that the registry tool will scan and record on the database. To make it easier we're going to use test fixtures from the registry codebase:
git clone --depth 1 -b master https://gitlab.com/gitlab-org/container-registry.git /tmp/fixtures cp -r /tmp/fixtures/registry/datastore/testdata/fixtures/importer/happy-path/docker /var/opt/gitlab/gitlab-rails/shared/registry
By doing the above we're populating the default storage location for the registry with a few sample repositories.
-
Reconfigure:
gitlab-ctl reconfigure
-
To test the newly added command, you need to replace/symlink the changed files in the right directory under your Omnibus installation. For example
ln -s /home/dev/files/gitlab-ctl-commands/registry_database.rb /opt/gitlab/embedded/service/omnibus-ctl/ ln -s /home/dev/files/gitlab-ctl-commands/lib/ /opt/gitlab/embedded/service/omnibus-ctl/
-
Now let's run all schema migrations to initialize our registry database:
$ gitlab-ctl registry-database migrate up <truncated> OK: applied 145 migrations in 19.781s
Command test
-
First we can test that while the registry is not in read-only mode, if we pass the
--dry-run
flag all should work:$ gitlab-ctl registry-database import --dry-run Running import Executing command: /opt/gitlab/embedded/bin/registry database import --dry-run /var/opt/gitlab/registry/config.yml 2023/11/24 15:33:24 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined INFO[0000] starting metadata import common_blobs=true dry_run=true environment=production go_version=go1.20.11 instance_id=9d7be951-d0fa-4ee8-8c4a-021963c84a68 pre_import=true repository_import=true service=registry version=v3.86.2-gitlab <truncated> INFO[0000] metadata import complete common_blobs=true dry_run=true duration_s=0.443187137 environment=production go_version=go1.20.11 instance_id=9d7be951-d0fa-4ee8-8c4a-021963c84a68 pre_import=true repository_import=true service=registry version=v3.86.2-gitlab
-
Then, we can test that the command is aborted in case the registry is not in read-only mode AND we pass one of the
-r/--all-repositories
or-2/--step-two
options:$ gitlab-ctl registry-database import --all-repositories Running import Command requires the container registry to be in read-only mode. Exiting...
-
Now update
/etc/gitlab/gitlab.rb
and setmaintenance.readonly.enabled
totrue
. Then reconfigure. -
Now we can repeat the process and it should succeed:
$ gitlab-ctl registry-database import --all-repositories Running import Executing command: /opt/gitlab/embedded/bin/registry database import --all-repositories /var/opt/gitlab/registry/config.yml 2023/11/24 15:35:35 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined INFO[0000] starting metadata import common_blobs=false dry_run=false environment=production go_version=go1.20.11 instance_id=c6ac7afd-f0cd-42c3-b10b-e6855c351c14 pre_import=false repository_import=true service=registry version=v3.86.2-gitlab <truncated> INFO[0000] metadata import complete common_blobs=false dry_run=false duration_s=0.553058922 environment=production go_version=go1.20.11 instance_id=c6ac7afd-f0cd-42c3-b10b-e6855c351c14 pre_import=false repository_import=true service=registry version=v3.86.2-gitlab
The remaining options can also be tested but they only differ in how the registry operates the import internally and all options are mapped directly with no modifications.
Related issues
Related to container-registry#1160 (closed).
Checklist
See Definition of done.
For anything in this list which will not be completed, please provide a reason in the MR discussion.
Required
-
MR title and description are up to date, accurate, and descriptive. -
MR targeting the appropriate branch. -
Latest Merge Result pipeline is green. -
When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow.
For GitLab team members
If you don't have access to this, the reviewer should trigger these jobs for you during the review process.
-
The manual Trigger:ee-package
jobs have a green pipeline running against latest commit. -
If config/software
orconfig/patches
directories are changed, make sure thebuild-package-on-all-os
job within theTrigger:ee-package
downstream pipeline succeeded. -
If you are changing anything SSL related, then the Trigger:package:fips
manual job within theTrigger:ee-package
downstream pipeline must succeed. -
If CI configuration is changed, the branch must be pushed to dev.gitlab.org
to confirm regular branch builds aren't broken.
Expected (please provide an explanation if not completing)
-
Test plan indicating conditions for success has been posted and passes. -
Documentation created/updated. -
Tests added. -
Integration tests added to GitLab QA. -
Equivalent MR/issue for the GitLab Chart opened. -
Validate potential values for new configuration settings. Formats such as integer 10
, duration10s
, URIscheme://user:passwd@host:port
may require quotation or other special handling when rendered in a template and written to a configuration file.
Merge request reports
Activity
changed milestone to %16.7
added devopssystems groupdistribution sectioncore platform labels
assigned to @jaime and @jdrpereira
added featureaddition headway-self-managed-registry-db typefeature workflowproduction + 1 deleted label
removed workflowproduction label
added workflowin dev label
- Resolved by Hossein Pursultani
14 Warnings fa856072: The commit body should not contain more than 72 characters per line. For more information, take a look at our Commit message guidelines. e5b8a10c: The commit subject must start with a capital letter. For more information, take a look at our Commit message guidelines. 5119b50e: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines. 5119b50e: The commit subject must start with a capital letter. For more information, take a look at our Commit message guidelines. d5f29669: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines. d5f29669: The commit subject must start with a capital letter. For more information, take a look at our Commit message guidelines. d5f29669: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines. a2703fdc: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines. a2703fdc: The commit subject must start with a capital letter. For more information, take a look at our Commit message guidelines. 8d227bab: The commit subject must start with a capital letter. For more information, take a look at our Commit message guidelines. 685ec082: The commit subject must start with a capital letter. For more information, take a look at our Commit message guidelines. dcfec7df: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines. dcfec7df: The commit subject must start with a capital letter. For more information, take a look at our Commit message guidelines. 522d6c5a: The commit subject must start with a capital letter. For more information, take a look at our Commit message guidelines. Reviewer roulette
Changes that require review have been detected! A merge request is normally reviewed by both a reviewer and a maintainer in its primary category and by a maintainer in all other categories.
To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.
To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.
Once you've decided who will review this merge request, mention them as you normally would! Danger does not automatically notify them for you.
Reviewer Maintainer @pursultani
(UTC+11, 11 hours ahead of
@jdrpereira
)@rmarshall
(UTC-5, 5 hours behind
@jdrpereira
)If needed, you can retry the
danger-review
job that generated this comment.Generated by
Danger
added 28 commits
-
7d37c3a8...34fdcdd4 - 20 commits from branch
master
- fa856072 - Add registry-database import command to gitlab-ctl
- baeaf36b - Fix help and command building
- 95e45886 - Add spec for import subcommand
- 4f05ab80 - Fix docs links
- f161af6d - Fix lint errors
- 9a0b3488 - Fix lint errors
- a38d85d4 - Add basic option specs
- 522d6c5a - fix spec for top import command
Toggle commit list-
7d37c3a8...34fdcdd4 - 20 commits from branch
mentioned in issue container-registry#1160 (closed)
added workflowready for review label and removed workflowin dev label
requested review from @pursultani
removed review request for @pursultani
requested review from @apatterson2
added workflowin review label and removed workflowready for review label
Triggered
Trigger:ee-package
in pipeline https://gitlab.com/gitlab-org/omnibus-gitlab/-/pipelines/1087729033.Allure report
allure-report-publisher
generated test report!qa-subset-test:
test report for e5b8a10cexpand test summary
+-----------------------------------------------------------------------+ | suites summary | +------------------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +------------------+--------+--------+---------+-------+-------+--------+ | Verify | 93 | 0 | 0 | 0 | 93 | ✅ | | Plan | 169 | 0 | 1 | 0 | 170 | ✅ | | Govern | 137 | 0 | 1 | 0 | 138 | ✅ | | Package | 124 | 0 | 6 | 0 | 130 | ✅ | | Framework sanity | 0 | 0 | 3 | 0 | 3 | ➖ | | Create | 199 | 1 | 28 | 0 | 228 | ❌ | | Monitor | 28 | 0 | 1 | 0 | 29 | ✅ | | Data Stores | 69 | 0 | 0 | 0 | 69 | ✅ | | Systems | 8 | 0 | 0 | 0 | 8 | ✅ | | Configure | 1 | 0 | 0 | 0 | 1 | ✅ | | GitLab Metrics | 2 | 0 | 1 | 0 | 3 | ✅ | | Manage | 2 | 0 | 0 | 0 | 2 | ✅ | +------------------+--------+--------+---------+-------+-------+--------+ | Total | 832 | 1 | 41 | 0 | 874 | ❌ | +------------------+--------+--------+---------+-------+-------+--------+
mentioned in epic &5521
- Resolved by João Pereira
- Resolved by João Pereira
- Resolved by João Pereira
LGTM expect for the help capitalization nit.
- Resolved by Balasankar 'Balu' C
Triggered
Trigger:ee-package
job in pipeline https://gitlab.com/gitlab-org/omnibus-gitlab/-/pipelines/1090739477.- Resolved by Andrew Patterson
Looks like help is broken:
sudo gitlab-ctl registry-database import --help /opt/gitlab/embedded/service/omnibus-ctl/lib/registry/import.rb:41:in `block in parse_options!': undefined local variable or method `opts' for Import:Module (NameError) opts.set_summary_width(SUMMARY_WIDTH) ^^^^ from /opt/gitlab/embedded/lib/ruby/3.1.0/optparse.rb:1576:in `block in parse_in_order' from /opt/gitlab/embedded/lib/ruby/3.1.0/optparse.rb:1559:in `catch' from /opt/gitlab/embedded/lib/ruby/3.1.0/optparse.rb:1559:in `parse_in_order' from /opt/gitlab/embedded/lib/ruby/3.1.0/optparse.rb:1553:in `order!' from /opt/gitlab/embedded/service/omnibus-ctl/lib/registry/import.rb:84:in `parse_options!' from /opt/gitlab/embedded/service/omnibus-ctl/lib/registry/registry_database.rb:71:in `block in populate_commands' from /opt/gitlab/embedded/lib/ruby/3.1.0/optparse.rb:1102:in `initialize' from /opt/gitlab/embedded/service/omnibus-ctl/lib/registry/registry_database.rb:70:in `new' from /opt/gitlab/embedded/service/omnibus-ctl/lib/registry/registry_database.rb:70:in `populate_commands' from /opt/gitlab/embedded/service/omnibus-ctl/lib/registry/registry_database.rb:45:in `parse_options!' from /opt/gitlab/embedded/service/omnibus-ctl/registry_database.rb:24:in `block in load_file' from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category' from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run' from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>' from /opt/gitlab/embedded/bin/omnibus-ctl:25:in `load' from /opt/gitlab/embedded/bin/omnibus-ctl:25:in `<main>'
LGTM:
-
Green pipeline at https://gitlab.com/gitlab-org/omnibus-gitlab/-/pipelines/1094289074. -
Code examination. -
Ran and help, checked for operation, English, format, and typos. -
Ran test plan shown above on VM. -
Ran each migrate option to ensure it is passed to/opt/gitlab/embedded/bin/registry database import
and the command did not blow up. I did not check the command/option functionality.
-
requested review from @balasankarc and removed review request for @apatterson2
mentioned in commit 5f15ec37
added workflowstaging-canary label and removed workflowin review label
added workflowcanary label and removed workflowstaging-canary label
added workflowstaging label and removed workflowcanary label
added workflowproduction label and removed workflowstaging label
added releasedcandidate label
mentioned in merge request !7327 (merged)
added Container Registry Self-Managed Rollout::BlocksOff by Default label and removed 1 deleted label
added releasedpublished label and removed releasedcandidate label
mentioned in epic gitlab-org#5521