Add log to stdout option to gitlab-ctl registry import command

What does this MR do?

Adds a new log to stdout option to the gitlab-ctl registry import command. This builds upon !7265 (merged). This new registry CLI option was introduced in container-registry!1519 (merged) and released in v3.88.0, which is the version used by Omnibus since !7311 (merged).

Syntax

$ gitlab-ctl registry-database import -h
Import filesystem metadata into the database

Usage:
  gitlab-ctl registry-database import [options]

Options:
  ...
  -l, --log-to-stdout               Write detailed log to standard output instead of showing progress bars

Testing

Initial setup

1. Setup a development Omnibus environment

2. Provision a separate PostgreSQL instance and create a logical database, for example:

   CREATE USER registry WITH PASSWORD 'registrypassword'
   CREATE DATABASE registry_omnibus WITH OWNER registry

   Alternatively, you can use the provisioned PostgreSQL instance but you will need to allow the registry to connect to it via Unix socket (sample)

3. Edit /etc/gitlab/gitlab.rb and enable the registry service:

   registry['enable'] = true

4. Also configure the database section under the registry configuration (update accordingly):

   registry['database'] = {
      'enabled' => true,
      'host' => '172.17.0.1',
      'port' => 5432,
      'user' => 'registry',
      'password' => 'registrypassword',
      'dbname' => 'registry_omnibus',
      'sslmode' => 'disable'
   }

5. Finally, update/add a storage section:

   registry['storage'] = {
     'filesystem' => {
       'rootdirectory' => "/var/opt/gitlab/gitlab-rails/shared/registry"
     }
   }

6. To be able to test the actual import we need some sample data that the registry tool will scan and record on the database. To make it easier we're going to use test fixtures from the registry codebase:

   git clone --depth 1 -b master https://gitlab.com/gitlab-org/container-registry.git /tmp/fixtures
   cp -r /tmp/fixtures/registry/datastore/testdata/fixtures/importer/happy-path/docker /var/opt/gitlab/gitlab-rails/shared/registry

   By doing the above we're populating the default storage location for the registry with a few sample repositories.

7. Reconfigure:

   gitlab-ctl reconfigure

8. To test the newly added command, you need to replace/symlink the changed files in the right directory under your Omnibus installation. For example

   ln -s /home/dev/files/gitlab-ctl-commands/registry_database.rb /opt/gitlab/embedded/service/omnibus-ctl/
   ln -s /home/dev/files/gitlab-ctl-commands/lib/ /opt/gitlab/embedded/service/omnibus-ctl/

9. Now let's run all schema migrations to initialize our registry database:

   $ gitlab-ctl registry-database migrate up

Command test

1. First we can test the import without the new flag:

    $ gitlab-ctl registry-database import
    Running import
    Executing command:
    /opt/gitlab/embedded/bin/registry database import /var/opt/gitlab/registry/config.yml
    |  [0s] registry import starting, detailed log written to: /root/omnibus-gitlab/container-registry/2024-01-04T14:55:35Z-registry-import.log
    |  [0s] step one: import manifests
    |  [2s] step two: import tags
    /  [2s] step three: import blobs
    /  [2s] registry import complete

2. Then, we can test the command with the new flag, and the output should be a plain log:

    $ gitlab-ctl registry-database import -l
    Running import
    Executing command:
    /opt/gitlab/embedded/bin/registry database import --log-to-stdout /var/opt/gitlab/registry/config.yml
    INFO[0000] starting metadata import                      common_blobs=true dry_run=false environment=production go_version=go1.21.0 instance_id=834080f2-007d-48c7-8f5a-47011243f3a1 pre_import=true repository_import=true service=registry version=015c499
    ...
    INFO[0002] metadata import complete                      common_blobs=true dry_run=false duration_s=2.526979383 environment=production go_version=go1.21.0 instance_id=834080f2-007d-48c7-8f5a-47011243f3a1 pre_import=true repository_import=true service=registry version=015c499

Related issues

Related to #8343 (closed).

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

• MR title and description are up to date, accurate, and descriptive.
• MR targeting the appropriate branch.
• Latest Merge Result pipeline is green.
• When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow.

For GitLab team members

If you don't have access to this, the reviewer should trigger these jobs for you during the review process.

• The manual Trigger:ee-package jobs have a green pipeline running against latest commit.
• If config/software or config/patches directories are changed, make sure the build-package-on-all-os job within the Trigger:ee-package downstream pipeline succeeded.
• If you are changing anything SSL related, then the Trigger:package:fips manual job within the Trigger:ee-package downstream pipeline must succeed.
• If CI configuration is changed, the branch must be pushed to dev.gitlab.org to confirm regular branch builds aren't broken.

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes.
• Documentation created/updated.
• Tests added.
• Integration tests added to GitLab QA.
• Equivalent MR/issue for the GitLab Chart opened.
• Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.