Backups not automatically setting "skipped" for disabled registries

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

Reproduced on 13.1.3-ee

If a backup is initiated on a GitLab instance that has the registry disabled, and then the backup file is restored on a GitLab instance that has the registry enabled, the restore fails with the following error:

❯ sudo gitlab-backup restore BACKUP=1595383295_2020_07_22_13.1.3-ee

Unpacking backup ... done
Before restoring the database, we will remove all existing
tables to avoid future upgrade problems. Be aware that if you have
custom tables in the GitLab database these tables and all data will be
removed.

Do you want to continue (yes/no)? yes
Removing all tables. Press `Ctrl-C` within 5 seconds to abort

............
stuff happens
............

rake aborted!
Errno::ENOENT: No such file or directory - /var/opt/gitlab/backups/registry.tar.gz
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/files.rb:76:in `run_pipeline!'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/files.rb:44:in `restore'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:245:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:79:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:registry:restore
(See full trace by running task with --trace)

Steps to reproduce

  1. Spin up a fresh GitLab instance with the following settings in gitlab.rb and run sudo gitlab-ctl reconfigure:

    gitlab_rails['registry_enabled'] = false

  2. Create a backup sudo gitlab-backup create. If you inspect the backup, notice that there is no registry.tar.gz. Also, the backup_information.yml doesn't have the skip flag for the registry:

    ❯ tar -tvf 1595383295_2020_07_22_13.1.3-ee_gitlab_backup.tar

drwx------ git/git           0 2020-07-22 02:01 repositories/
drwxr-xr-x git/git           0 2020-07-22 02:01 repositories/@hashed/
drwxr-xr-x git/git           0 2020-07-22 02:01 repositories/@hashed/6b/
drwxr-xr-x git/git           0 2020-07-22 02:01 repositories/@hashed/6b/86/
-rw-r--r-- git/git       48536 2020-07-22 02:01 repositories/@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.bundle
drwxr-xr-x git/git           0 2020-07-22 02:01 repositories/@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b/
drwxr-xr-x git/git           0 2020-07-22 02:01 db/
-rw------- git/git      171515 2020-07-22 02:01 db/database.sql.gz
-rw------- git/git         275 2020-07-22 02:01 uploads.tar.gz
-rw------- git/git         151 2020-07-22 02:01 builds.tar.gz
-rw------- git/git         151 2020-07-22 02:01 artifacts.tar.gz
-rw------- git/git         155 2020-07-22 02:01 pages.tar.gz
-rw------- git/git         150 2020-07-22 02:01 lfs.tar.gz
-rw-r--r-- git/git         193 2020-07-22 02:01 backup_information.yml

❯ tar -axf 1595383295_2020_07_22_13.1.3-ee_gitlab_backup.tar backup_information.yml -O

:db_version: '20200615232735'
:backup_created_at: 2020-07-22 02:01:35.003329994 +00:00
:gitlab_version: 13.1.3-ee
:tar_version: tar (GNU tar) 1.28
:installation_type: omnibus-gitlab
:skipped:

  3. Restore the backup sudo gitlab-backup restore BACKUP=1595383295_2020_07_22_13.1.3-ee. This should be successful and confirms the backup is good.

  4. Let's now reproduce the bug. Change the gitlab.rb to enable the registry and and sudo gitlab-ctl reconfigure:

    gitlab_rails['registry_enabled'] = true

  5. Try restoring the backup again sudo gitlab-backup restore BACKUP=1595383295_2020_07_22_13.1.3-ee.

    Note: If you see Errno::ENOENT: No such file or directory @ realpath_rec - /var/opt/gitlab/gitlab-rails/shared/registry, run the following, and then run the restore again.

       mkdir /var/opt/gitlab/gitlab-rails/shared/registry
   chown git:git /var/opt/gitlab/gitlab-rails/shared/registry

    Finally, you should see this error:

    Errno::ENOENT: No such file or directory - /var/opt/gitlab/backups/registry.tar.gz

Example Project

(If possible, please create an example project here on GitLab.com that exhibits the problematic behavior, and link to it here in the bug report)

(If you are using an older version of GitLab, this will also determine whether the bug is fixed in a more recent version)

What is the current bug behavior?

A GitLab instance with a disabled registry does not add the SKIP registry flag to the backup_information.yml file. As a result, when restoring the backup on a GitLab instance with an enabled registry, the restore operation fails.

What is the expected correct behavior?

A GitLab instance with a disabled registry should add the SKIP registry flag to the backup_information.yml file.

Relevant logs and/or screenshots

(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's tough to read otherwise.)

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to GitLab environment info 

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)

Workarounds

  • Use sudo gitlab-backup create SKIP=registry when backing up GitLab on a system with the registry disabled. The restore will then work as expected.

  • Add :skipped: registry to the backup_information.yml file inside a backup tar. The restore will then work as expected.

    # From
:db_version: '20200615232735'
:backup_created_at: 2020-07-22 02:01:35.003329994 +00:00
:gitlab_version: 13.1.3-ee
:tar_version: tar (GNU tar) 1.28
:installation_type: omnibus-gitlab
:skipped:

# To:
:db_version: '20200615232735'
:backup_created_at: 2020-07-22 02:01:35.003329994 +00:00
:gitlab_version: 13.1.3-ee
:tar_version: tar (GNU tar) 1.28
:installation_type: omnibus-gitlab
:skipped: registry
Edited by 🤖 GitLab Bot 🤖