`gitlab-backup restore` causes subsequent backups to fail
Summary
On a fresh omnibus GitLab installation, running gitlab-backup restore
causes subsequent run of gitlab-backup create
to fail.
Steps to reproduce
- Install a fresh GitLab server
- Run
gitlab-backup create
- Run
gitlab-backup restore
- Run
gitlab-backup create
What is the current bug behavior?
The last command fails.
What is the expected correct behavior?
That the command doesn't fail.
Relevant logs and/or screenshots
root@gitlab1:~# gitlab-backup create
2020-01-24 10:07:06 +0000 -- Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2020-01-24 10:07:07 +0000 -- done
2020-01-24 10:07:07 +0000 -- Dumping repositories ...
2020-01-24 10:07:07 +0000 -- done
2020-01-24 10:07:07 +0000 -- Dumping uploads ...
2020-01-24 10:07:07 +0000 -- done
2020-01-24 10:07:07 +0000 -- Dumping builds ...
2020-01-24 10:07:07 +0000 -- done
2020-01-24 10:07:07 +0000 -- Dumping artifacts ...
2020-01-24 10:07:07 +0000 -- done
2020-01-24 10:07:07 +0000 -- Dumping pages ...
2020-01-24 10:07:07 +0000 -- done
2020-01-24 10:07:07 +0000 -- Dumping lfs objects ...
2020-01-24 10:07:07 +0000 -- done
2020-01-24 10:07:07 +0000 -- Dumping container registry images ...
2020-01-24 10:07:07 +0000 -- done
Creating backup archive: 1579860427_2020_01_24_12.7.0-ee_gitlab_backup.tar ... done
Uploading backup archive to remote storage ... skipped
Deleting tmp directories ... done
done
done
done
done
done
done
done
done
Deleting old backups ... done. (0 removed)
Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data
and are not included in this backup. You will need these files to restore a backup.
Please back them up manually.
Backup task is done.
root@gitlab1:~# gitlab-backup restore
Transfering ownership of /var/opt/gitlab/gitlab-rails/shared/registry to git:git
Unpacking backup ... done
Before restoring the database, we will remove all existing
tables to avoid future upgrade problems. Be aware that if you have
custom tables in the GitLab database these tables and all data will be
removed.
Do you want to continue (yes/no)? yes
(I've cut out all the boring output from restoring the database, if there's anything interesting right before that scrolled of my screen as a result of all that output, that is cut too, but as the problem doesn't occur with this command, I think it's fine, if you really need it, ask, and I'll find a way to get it.
WARNING: no privileges were granted for "public"
GRANT
[DONE]
2020-01-24 10:10:27 +0000 -- done
2020-01-24 10:10:27 +0000 -- Restoring repositories ...
2020-01-24 10:10:27 +0000 -- done
2020-01-24 10:10:27 +0000 -- Restoring uploads ...
2020-01-24 10:10:27 +0000 -- done
2020-01-24 10:10:27 +0000 -- Restoring builds ...
2020-01-24 10:10:27 +0000 -- done
2020-01-24 10:10:27 +0000 -- Restoring artifacts ...
2020-01-24 10:10:27 +0000 -- done
2020-01-24 10:10:27 +0000 -- Restoring pages ...
2020-01-24 10:10:27 +0000 -- done
2020-01-24 10:10:27 +0000 -- Restoring lfs objects ...
2020-01-24 10:10:27 +0000 -- done
2020-01-24 10:10:27 +0000 -- Restoring container registry images ...
2020-01-24 10:10:27 +0000 -- done
This task will now rebuild the authorized_keys file.
You will lose any data stored in the authorized_keys file.
Do you want to continue (yes/no)? yes
Deleting tmp directories ... done
done
done
done
done
done
done
done
done
Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data
and are not included in this backup. You will need to restore these files manually.
Restore task is done.
Transfering ownership of /var/opt/gitlab/gitlab-rails/shared/registry to registry:registry
root@gitlab1:~# gitlab-backup create
2020-01-24 10:13:54 +0000 -- Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2020-01-24 10:13:55 +0000 -- done
2020-01-24 10:13:55 +0000 -- Dumping repositories ...
2020-01-24 10:13:55 +0000 -- done
2020-01-24 10:13:55 +0000 -- Dumping uploads ...
2020-01-24 10:13:55 +0000 -- done
2020-01-24 10:13:55 +0000 -- Dumping builds ...
2020-01-24 10:13:55 +0000 -- done
2020-01-24 10:13:55 +0000 -- Dumping artifacts ...
2020-01-24 10:13:55 +0000 -- done
2020-01-24 10:13:55 +0000 -- Dumping pages ...
2020-01-24 10:13:55 +0000 -- done
2020-01-24 10:13:55 +0000 -- Dumping lfs objects ...
2020-01-24 10:13:55 +0000 -- done
2020-01-24 10:13:55 +0000 -- Dumping container registry images ...
rake aborted!
Backup::Error: Backup failed
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/files.rb:81:in `run_pipeline!'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/files.rb:37:in `dump'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:222:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:17:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:registry:create
(See full trace by running task with --trace)
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
The output of: sudo gitlab-rake gitlab:env:info
System information
System: Ubuntu 16.04
Proxy: no
Current User: git
Using RVM: no
Ruby Version: 2.6.5p114
Gem Version: 2.7.10
Bundler Version:1.17.3
Rake Version: 12.3.3
Redis Version: 5.0.7
Git Version: 2.24.1
Sidekiq Version:5.2.7
Go Version: unknown
GitLab information
Version: 12.7.0-ee
Revision: c6ca82f7978
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 10.9
URL: https://gitlab-kitchen.one.com
HTTP Clone URL: https://gitlab-kitchen.one.com/some-group/some-project.git
SSH Clone URL: git@gitlab-kitchen.one.com:some-group/some-project.git
Elasticsearch: no
Geo: no
Using LDAP: no
Using Omniauth: yes
Omniauth Providers: github
GitLab Shell
Version: 11.0.0
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)
Possible fixes
There does not seem to be any problem if using the rake tasks (for both backup and restore) directly, so perhaps this is related to the chmod
's the wrapper scripts does when restoring?