Backup registry failed

Summary

Try backup GitLab 11.8.1. Backup failed on "Dumping container registry images" stage.

Update without backup. And backup failed on 11.8.2 version too.

Steps to reproduce

I think registry backup fail after i use docker-distribution-pruner for cleaning registry

docker exec -it -e EXPERIMENTAL=true gitlab docker-distribution-pruner -delete -config=/var/opt/gitlab/registry/config.yml

docker exec -t gitlab gitlab-rake gitlab:backup:create

What is the current bug behavior?

Failed backup registry

What is the expected correct behavior?

Success backup registry

Relevant logs and/or screenshots

** Invoke gitlab:backup:registry:create (first_time)
** Invoke gitlab_environment
** Execute gitlab:backup:registry:create
2019-03-19 08:20:02 +0000 -- Dumping container registry images ...
rake aborted!
Backup::Error: Backup failed
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/files.rb:81:in `run_pipeline!'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/files.rb:37:in `dump'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:214:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:194:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:183:in `invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:17:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:194:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:183:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli.rb:424:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:registry:create

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:
Current User:	git
Using RVM:	no
Ruby Version:	2.5.3p105
Gem Version:	2.7.6
Bundler Version:1.16.6
Rake Version:	12.3.2
Redis Version:	3.2.12
Git Version:	2.18.1
Sidekiq Version:5.2.5
Go Version:	unknown
GitLab information
Version:	11.8.2
Revision:	b1268e9
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
URL:		https://gitlab
HTTP Clone URL:	https://gitlab/some-group/some-project.git
SSH Clone URL:	git@gitlab:some-group/some-project.git
Using LDAP:	yes
Using Omniauth:	yes
Omniauth Providers:
GitLab Shell
Version:	8.4.4
Repository storage paths:

• default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 8.4.4 ? ... OK (8.4.4)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
...
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
3/1 ... yes
9/4 ... yes
6/5 ... yes
9/6 ... yes
3/7 ... yes
6/8 ... yes
3/9 ... yes
6/10 ... yes
3/11 ... yes
3/12 ... yes
14/13 ... yes
14/15 ... yes
14/16 ... yes
14/17 ... yes
14/18 ... yes
14/19 ... yes
14/20 ... yes
14/21 ... yes
14/22 ... yes
14/23 ... yes
14/24 ... yes
14/25 ... yes
14/26 ... yes
14/27 ... yes
9/28 ... yes
14/29 ... yes
14/32 ... yes
14/33 ... yes
14/34 ... yes
25/35 ... yes
25/37 ... yes
3/38 ... yes
3/40 ... yes
14/42 ... yes
14/43 ... yes
9/46 ... yes
38/53 ... yes
14/54 ... yes
14/55 ... yes
14/56 ... yes
14/58 ... yes
47/59 ... yes
14/60 ... yes
47/61 ... yes
47/62 ... yes
47/63 ... yes
47/64 ... yes
47/65 ... yes
47/66 ... yes
47/67 ... yes
47/68 ... yes
47/69 ... yes
47/70 ... yes
47/71 ... yes
47/72 ... yes
14/73 ... yes
14/75 ... yes
14/79 ... yes
3/80 ... yes
60/81 ... yes
14/82 ... yes
14/83 ... yes
14/84 ... yes
14/85 ... yes
61/86 ... yes
61/87 ... yes
3/88 ... yes
14/89 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.5.3)
Git version >= 2.18.0 ? ... yes (2.18.1)
Git user has default SSH configuration? ... yes
Active users: ... 28
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Additional Information

docker info

Containers: 6
 Running: 4
 Paused: 0
 Stopped: 2
Images: 22
Server Version: 17.03.1-ce
Storage Driver: devicemapper
 Pool Name: docker-253:0-100663398-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 9.986 GB
 Data Space Total: 107.4 GB
 Data Space Available: 22.52 GB
 Metadata Space Used: 9.814 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.138 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.140-RHEL7 (2017-05-03)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-693.21.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 13.7 GiB
Name: gitlab
ID: YTHW:JGKL:DIVA:XKBT:MTGZ:J7R5:IICD:3GRH:SQVA:FFBP:UIZ6:CS73
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false