Add registry-database import command to gitlab-ctl

What does this MR do?

Adds a new import command under gitlab-ctl registry-database that wraps the registry database import tool, which provides users with the ability to migrate from filesystem metadata (old registry) to database metadata (new registry).

Syntax

$ gitlab-ctl registry-database import -h
Import filesystem metadata into the database

Usage:
  gitlab-ctl registry-database import [options]

Options:
  -B, --common-blobs                import all blob metadata from common storage
  -c, --row-count                   count and log number of rows across relevant database tables on (pre)import completion
  -d, --dry-run                     do not commit changes to the database
  -e, --require-empty-database      abort import if the database is not empty
  -p, --pre-import                  import immutable repository-scoped data to speed up a following import
  -r, --all-repositories            import all repository-scoped data
  -h, --help                        help for import
  -1, --step-one pre-import         perform step one of a multi-step import: alias for pre-import
  -2, --step-two all-repositories   perform step two of a multi-step import: alias for all-repositories
  -3, --step-three common-blobs     perform step three of a multi-step import: alias for common-blobs

For more details about each option please see the documentation linked above.

Validations

1. The registry must be enabled before proceeding
2. The registry binary exists
3. The configuration file exists
4. If the registry is running and not set to read-only mode, the execution is aborted when using the -r/--all-repositories or -2/--step-two options

Why?

As of today, users in the Beta program of Release container registry with metadata databa... (&5521) need to bypass gitlab-ctl and interact directly with the registry CLI to perform a migration. With this change we allow them to keep using gitlab-ctl, while also providing another layer for basic validations.

Testing

Initial setup

1. Setup a development Omnibus environment

2. Provision a separate PostgreSQL instance and create a logical database, for example:

   CREATE USER registry WITH PASSWORD 'registrypassword'
   CREATE DATABASE registry_omnibus WITH OWNER registry

   Alternatively, you can use the provisioned PostgreSQL instance but you will need to allow the registry to connect to it via Unix socket (sample)

3. Edit /etc/gitlab/gitlab.rb and enable the registry service:

   registry['enable'] = true

4. Also configure the database section under the registry configuration (update accordingly):

   registry['database'] = {
      'enabled' => true,
      'host' => '172.17.0.1',
      'port' => 5432,
      'user' => 'registry',
      'password' => 'registrypassword',
      'dbname' => 'registry_omnibus',
      'sslmode' => 'disable'
   }

5. Finally, update/add a storage section to include maintenance.readonly.enabled. Let it set to false for now:

   registry['storage'] = {
     'filesystem' => {
       'rootdirectory' => "/var/opt/gitlab/gitlab-rails/shared/registry"
     },
     'maintenance' => {
       'readonly' => {
         'enabled' => false
       }
     }
   }

6. To be able to test the actual import we need some sample data that the registry tool will scan and record on the database. To make it easier we're going to use test fixtures from the registry codebase:

   git clone --depth 1 -b master https://gitlab.com/gitlab-org/container-registry.git /tmp/fixtures
   cp -r /tmp/fixtures/registry/datastore/testdata/fixtures/importer/happy-path/docker /var/opt/gitlab/gitlab-rails/shared/registry

   By doing the above we're populating the default storage location for the registry with a few sample repositories.

7. Reconfigure:

   gitlab-ctl reconfigure

8. To test the newly added command, you need to replace/symlink the changed files in the right directory under your Omnibus installation. For example

   ln -s /home/dev/files/gitlab-ctl-commands/registry_database.rb /opt/gitlab/embedded/service/omnibus-ctl/
   ln -s /home/dev/files/gitlab-ctl-commands/lib/ /opt/gitlab/embedded/service/omnibus-ctl/

9. Now let's run all schema migrations to initialize our registry database:

   $ gitlab-ctl registry-database migrate up
   <truncated>
   OK: applied 145 migrations in 19.781s

Command test

1. First we can test that while the registry is not in read-only mode, if we pass the --dry-run flag all should work:

   $ gitlab-ctl registry-database import --dry-run
   Running import
   Executing command:
   /opt/gitlab/embedded/bin/registry database import --dry-run /var/opt/gitlab/registry/config.yml
   2023/11/24 15:33:24 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
   INFO[0000] starting metadata import                      common_blobs=true dry_run=true environment=production go_version=go1.20.11 instance_id=9d7be951-d0fa-4ee8-8c4a-021963c84a68 pre_import=true repository_import=true service=registry version=v3.86.2-gitlab
   <truncated>
   INFO[0000] metadata import complete                      common_blobs=true dry_run=true duration_s=0.443187137 environment=production go_version=go1.20.11 instance_id=9d7be951-d0fa-4ee8-8c4a-021963c84a68 pre_import=true repository_import=true service=registry version=v3.86.2-gitlab

2. Then, we can test that the command is aborted in case the registry is not in read-only mode AND we pass one of the -r/--all-repositories or -2/--step-two options:

   $ gitlab-ctl registry-database import --all-repositories
   Running import
   Command requires the container registry to be in read-only mode. Exiting...

3. Now update /etc/gitlab/gitlab.rb and set maintenance.readonly.enabled to true. Then reconfigure.

4. Now we can repeat the process and it should succeed:

   $ gitlab-ctl registry-database import --all-repositories
   Running import
   Executing command:
   /opt/gitlab/embedded/bin/registry database import --all-repositories /var/opt/gitlab/registry/config.yml
   2023/11/24 15:35:35 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
   INFO[0000] starting metadata import                      common_blobs=false dry_run=false environment=production go_version=go1.20.11 instance_id=c6ac7afd-f0cd-42c3-b10b-e6855c351c14 pre_import=false repository_import=true service=registry version=v3.86.2-gitlab
   <truncated>
   INFO[0000] metadata import complete                      common_blobs=false dry_run=false duration_s=0.553058922 environment=production go_version=go1.20.11 instance_id=c6ac7afd-f0cd-42c3-b10b-e6855c351c14 pre_import=false repository_import=true service=registry version=v3.86.2-gitlab

   The remaining options can also be tested but they only differ in how the registry operates the import internally and all options are mapped directly with no modifications.

Related issues

Related to container-registry#1160 (closed).

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

• MR title and description are up to date, accurate, and descriptive.
• MR targeting the appropriate branch.
• Latest Merge Result pipeline is green.
• When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow.

For GitLab team members

If you don't have access to this, the reviewer should trigger these jobs for you during the review process.

• The manual Trigger:ee-package jobs have a green pipeline running against latest commit.
• If config/software or config/patches directories are changed, make sure the build-package-on-all-os job within the Trigger:ee-package downstream pipeline succeeded.
• If you are changing anything SSL related, then the Trigger:package:fips manual job within the Trigger:ee-package downstream pipeline must succeed.
• If CI configuration is changed, the branch must be pushed to dev.gitlab.org to confirm regular branch builds aren't broken.

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes.
• Documentation created/updated.
• Tests added.
• Integration tests added to GitLab QA.
• Equivalent MR/issue for the GitLab Chart opened.
• Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.

files/gitlab-ctl-commands/lib/registry/import.rb

+require 'optparse'
+
+module Import
+  CMD_NAME = 'import'.freeze
+  SUMMARY_WIDTH = 40
+  DESC_INDENT = 45
+
+  def self.indent(str, len)
+    str.gsub(/%%/, ' ' * len)
+  end
+
+  USAGE = <<~EOS.freeze
+  Import filesystem metadata into the database
+
+  See documentation at https://gitlab.com/gitlab-org/container-registry/-/blob/master/docs/database-import-tool.md
+
+  Usage:
+    gitlab-ctl registry-database import [options]
+
+  Options:
+    -B, --common-blobs                Import all blob metadata from common storage
+    -c, --row-count                   Count and log number of rows across relevant database tables on (pre)import completion
+    -d, --dry-run                     Do not commit changes to the database
+    -e, --require-empty-database      Abort import if the database is not empty
+    -p, --pre-import                  Import immutable repository-scoped data to speed up a following import
+    -r, --all-repositories            Import all repository-scoped data
+    -h, --help                        Help for import
+    -1, --step-one pre-import         Perform step one of a multi-step import: alias for pre-import
+    -2, --step-two all-repositories   Perform step two of a multi-step import: alias for all-repositories
+    -3, --step-three common-blobs     Perform step three of a multi-step import: alias for common-blobs
+  EOS
+
+  def self.parse_options!(args, parser, options)
+    return unless args.include? CMD_NAME
+
+    loop do
+      break if args.shift == CMD_NAME
+    end
+
+    parser.on('-h', '--help', 'Usage help') do
+      parser.set_summary_width(SUMMARY_WIDTH)
+      Kernel.puts USAGE
+      Kernel.exit 0
+    end
+
+    parser.on('-B', '--common-blobs', indent('import all blob metadata from common storage', DESC_INDENT)) do
+      options[:common_blobs] = '--common-blobs'
+    end
+
+    parser.on('-c', '--row-count', indent('count and log number of rows across relevant database tables on (pre)import completion', DESC_INDENT)) do
+      options[:row_count] = '--row-count'
+    end
+
+    parser.on('-d', '--dry-run', indent('do not commit changes to the database', DESC_INDENT)) do
+      options[:dry_run] = '--dry-run'
+    end
+
+    parser.on('-e', '--require-empty-database', indent('abort import if the database is not empty', DESC_INDENT)) do
+      options[:empty] = '--require-empty-database'
+    end
+
+    parser.on('-p', '--pre-import', indent('import immutable repository-scoped data to speed up a following import', DESC_INDENT)) do
+      options[:pre_import] = '--pre-import'
+    end
+
+    parser.on('-r', '--all-repositories', indent('import all repository-scoped data', DESC_INDENT)) do
+      options[:all_repositories] = '--all-repositories'
+      options[:needs_read_only] = true
+    end
+
+    parser.on('-1', '--step-one', indent('perform step one of a multi-step import: alias for pre-import', DESC_INDENT)) do
+      options[:step_one] = '--step-one'
+    end
+
+    parser.on('-2', '--step-two', indent('perform step two of a multi-step import: alias for all-repositories', DESC_INDENT)) do
+      options[:step_two] = '--step-two'
+      options[:needs_read_only] = true
+    end
+
+    parser.on('-3', '--step-three', indent('perform step three of a multi-step import: alias for common-blobs', DESC_INDENT)) do
+      options[:step_three] = '--step-three'
+    end
+
+    parser.order!(args)
+
+    options
+  end
+end