Skip to content

Resolve Docker garbage collection issues for GitLab Container Registries hosted on Digital Ocean S3.

Summary

There is a known docker issue, where Ceph based S3 will drop all requests with absolute URI, impacting the Docker garbage collection process. Since Digital Ocean uses Ceph, it has prevented users from running garbage collection, if they are using Digital Ocean S3 for storage.

Steps to reproduce

Define a space at Digital Ocean and use it as the storage for your GitLab Container Registry. Delete tags and attempt to run garbage collection

  • sudo gitlab-ctl registry-garbage-collect

What is the current bug behavior?

Trying to run gitlab-ctl registry-garbage-collect fails with a SIGSEGV in docker's registry command

What is the expected correct behavior?

registry-garbage-collect works without failing on a SIGSEGV

Relevant logs and/or screenshots

gitlab-ctl registry-garbage-collect
ok: down: registry: 1s, normally up
Running garbage-collect using configuration ["/opt/gitlab/embedded/bin/registry", "garbage-collect", "/var/opt/gitlab/registry/config.yml"], this might take a while...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xc5965d]

goroutine 1 [running]:
github.com/docker/distribution/registry/storage/driver/s3-aws.(*driver).doWalk.func1(0xc000143a00, 0xc00044e601, 0xc0001fd6e8)
	/var/cache/omnibus/src/registry/src/github.com/docker/distribution/registry/storage/driver/s3-aws/s3.go:973 +0x9d
github.com/docker/distribution/vendor/github.com/aws/aws-sdk-go/service/s3.(*S3).ListObjectsV2PagesWithContext(0xc00013e430, 0x7f40081eb270, 0xc00045e380, 0xc000334af0, 0xc0001857e8, 0x0, 0x0, 0x0, 0x1, 0x2)
	/var/cache/omnibus/src/registry/src/github.com/docker/distribution/vendor/github.com/aws/aws-sdk-go/service/s3/api.go:4198 +0x111
github.com/docker/distribution/registry/storage/driver/s3-aws.(*driver).doWalk(0xc000143980, 0xfff4a0, 0xc00045e310, 0xc0001fd8a8, 0xc000354e71, 0x20, 0xeb34f5, 0x1, 0xc00044b4a0, 0x0, ...)
	/var/cache/omnibus/src/registry/src/github.com/docker/distribution/registry/storage/driver/s3-aws/s3.go:971 +0x3a1
github.com/docker/distribution/registry/storage/driver/s3-aws.(*driver).Walk(0xc000143980, 0xfff4a0, 0xc00045e310, 0xc0004620c0, 0x20, 0xc00044b4a0, 0x2, 0x0)
	/var/cache/omnibus/src/registry/src/github.com/docker/distribution/registry/storage/driver/s3-aws/s3.go:919 +0x160
github.com/docker/distribution/registry/storage/driver/base.(*Base).Walk(0xc000430cc0, 0xfff4a0, 0xc00045e310, 0xc0004620c0, 0x20, 0xc00044b4a0, 0x0, 0x0)
	/var/cache/omnibus/src/registry/src/github.com/docker/distribution/registry/storage/driver/base/base.go:239 +0x234
github.com/docker/distribution/registry/storage.(*registry).Enumerate(0xc00045e2a0, 0xfff1e0, 0xc000433080, 0xc00044e580, 0x0, 0x0)
	/var/cache/omnibus/src/registry/src/github.com/docker/distribution/registry/storage/catalog.go:67 +0xe3
github.com/docker/distribution/registry/storage.MarkAndSweep(0xfff1e0, 0xc000433080, 0x100b340, 0xc000430cc0, 0x10013a0, 0xc00045e2a0, 0x0, 0x10013a0, 0xc00045e2a0)
	/var/cache/omnibus/src/registry/src/github.com/docker/distribution/registry/storage/garbagecollect.go:40 +0x1e7
github.com/docker/distribution/registry.glob..func3(0x1733fc0, 0xc0004302b0, 0x1, 0x1)
	/var/cache/omnibus/src/registry/src/github.com/docker/distribution/registry/root.go:80 +0x48a
github.com/docker/distribution/vendor/github.com/spf13/cobra.(*Command).execute(0x1733fc0, 0xc000430260, 0x1, 0x1, 0x1733fc0, 0xc000430260)
	/var/cache/omnibus/src/registry/src/github.com/docker/distribution/vendor/github.com/spf13/cobra/command.go:495 +0x191
github.com/docker/distribution/vendor/github.com/spf13/cobra.(*Command).Execute(0x1734300, 0xc0001fdf88, 0xc000086058)
	/var/cache/omnibus/src/registry/src/github.com/docker/distribution/vendor/github.com/spf13/cobra/command.go:560 +0x2f4
main.main()
	/var/cache/omnibus/src/registry/src/github.com/docker/distribution/cmd/registry/main.go:23 +0x2d

Failed to run garbage-collect command, starting registry service.
ok: run: registry: (pid 20249) 1s

Results of GitLab environment info

Expand for output related to GitLab environment info 
root@gitlab:~#  gitlab-rake gitlab:env:info


System information
System:		Ubuntu 18.04
Current User:	git
Using RVM:	no
Ruby Version:	2.6.3p62
Gem Version:	2.7.9
Bundler Version:1.17.3
Rake Version:	12.3.2
Redis Version:	3.2.12
Git Version:	2.21.0
Sidekiq Version:5.2.7
Go Version:	unknown


GitLab information
Version:	12.0.3
Revision:	08a51a9db93
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	10.7
URL:		https://gitlab.noumenadigital.com
HTTP Clone URL:	https://gitlab.noumenadigital.com/some-group/some-project.git
SSH Clone URL:	git@gitlab.noumenadigital.com:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: saml


GitLab Shell
Version:	9.3.0
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
root@gitlab:~# sudo gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...


Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 9.3.0 ? ... OK (9.3.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
3/1 ... yes
3/2 ... yes
3/3 ... yes
3/4 ... yes
3/5 ... yes
3/7 ... yes
10/8 ... yes
10/9 ... yes
10/10 ... yes
1/11 ... yes
3/12 ... yes
10/13 ... yes
14/14 ... yes
14/16 ... yes
10/17 ... yes
3/20 ... yes
3/21 ... yes
18/22 ... yes
3/23 ... yes
3/24 ... yes
3/25 ... yes
3/26 ... yes
10/27 ... yes
22/28 ... yes
13/29 ... yes
3/30 ... yes
3/31 ... yes
10/32 ... yes
3/33 ... yes
1/34 ... yes
25/35 ... yes
18/36 ... yes
3/37 ... yes
37/38 ... yes
3/40 ... yes
39/41 ... yes
3/42 ... yes
53/43 ... yes
53/44 ... yes
53/45 ... yes
53/46 ... yes
53/47 ... yes
3/50 ... yes
10/51 ... yes
3/52 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.3)
Git version >= 2.21.0 ? ... yes (2.21.0)
Git user has default SSH configuration? ... yes
Active users: ... 29


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished
Edited by Tim Rizzi