gitlab-ctl registry-garbage-collect fails with SIGSEGV on Digital Ocean S3.

Summary

Due to a known docker issue, registry garbage collection fails for Digital Ocean S3

https://github.com/docker/distribution/issues/2695

Steps to reproduce

Define a space at digital ocean and use this as a docker registry

Example Project

(If possible, please create an example project here on GitLab.com that exhibits the problematic behavior, and link to it here in the bug report)

(If you are using an older version of GitLab, this will also determine whether the bug is fixed in a more recent version)

What is the current bug behavior?

Trying to run gitlab-ctl registry-garbage-collect fails with a SIGSEGV in docker's registry command

What is the expected correct behavior?

registry-garbage-collect works without failing on a SIGSEGV

Relevant logs and/or screenshots

gitlab-ctl registry-garbage-collect
ok: down: registry: 1s, normally up
Running garbage-collect using configuration ["/opt/gitlab/embedded/bin/registry", "garbage-collect", "/var/opt/gitlab/registry/config.yml"], this might take a while...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xc5965d]

goroutine 1 [running]:
github.com/docker/distribution/registry/storage/driver/s3-aws.(*driver).doWalk.func1(0xc000143a00, 0xc00044e601, 0xc0001fd6e8)
	/var/cache/omnibus/src/registry/src/github.com/docker/distribution/registry/storage/driver/s3-aws/s3.go:973 +0x9d
github.com/docker/distribution/vendor/github.com/aws/aws-sdk-go/service/s3.(*S3).ListObjectsV2PagesWithContext(0xc00013e430, 0x7f40081eb270, 0xc00045e380, 0xc000334af0, 0xc0001857e8, 0x0, 0x0, 0x0, 0x1, 0x2)
	/var/cache/omnibus/src/registry/src/github.com/docker/distribution/vendor/github.com/aws/aws-sdk-go/service/s3/api.go:4198 +0x111
github.com/docker/distribution/registry/storage/driver/s3-aws.(*driver).doWalk(0xc000143980, 0xfff4a0, 0xc00045e310, 0xc0001fd8a8, 0xc000354e71, 0x20, 0xeb34f5, 0x1, 0xc00044b4a0, 0x0, ...)
	/var/cache/omnibus/src/registry/src/github.com/docker/distribution/registry/storage/driver/s3-aws/s3.go:971 +0x3a1
github.com/docker/distribution/registry/storage/driver/s3-aws.(*driver).Walk(0xc000143980, 0xfff4a0, 0xc00045e310, 0xc0004620c0, 0x20, 0xc00044b4a0, 0x2, 0x0)
	/var/cache/omnibus/src/registry/src/github.com/docker/distribution/registry/storage/driver/s3-aws/s3.go:919 +0x160
github.com/docker/distribution/registry/storage/driver/base.(*Base).Walk(0xc000430cc0, 0xfff4a0, 0xc00045e310, 0xc0004620c0, 0x20, 0xc00044b4a0, 0x0, 0x0)
	/var/cache/omnibus/src/registry/src/github.com/docker/distribution/registry/storage/driver/base/base.go:239 +0x234
github.com/docker/distribution/registry/storage.(*registry).Enumerate(0xc00045e2a0, 0xfff1e0, 0xc000433080, 0xc00044e580, 0x0, 0x0)
	/var/cache/omnibus/src/registry/src/github.com/docker/distribution/registry/storage/catalog.go:67 +0xe3
github.com/docker/distribution/registry/storage.MarkAndSweep(0xfff1e0, 0xc000433080, 0x100b340, 0xc000430cc0, 0x10013a0, 0xc00045e2a0, 0x0, 0x10013a0, 0xc00045e2a0)
	/var/cache/omnibus/src/registry/src/github.com/docker/distribution/registry/storage/garbagecollect.go:40 +0x1e7
github.com/docker/distribution/registry.glob..func3(0x1733fc0, 0xc0004302b0, 0x1, 0x1)
	/var/cache/omnibus/src/registry/src/github.com/docker/distribution/registry/root.go:80 +0x48a
github.com/docker/distribution/vendor/github.com/spf13/cobra.(*Command).execute(0x1733fc0, 0xc000430260, 0x1, 0x1, 0x1733fc0, 0xc000430260)
	/var/cache/omnibus/src/registry/src/github.com/docker/distribution/vendor/github.com/spf13/cobra/command.go:495 +0x191
github.com/docker/distribution/vendor/github.com/spf13/cobra.(*Command).Execute(0x1734300, 0xc0001fdf88, 0xc000086058)
	/var/cache/omnibus/src/registry/src/github.com/docker/distribution/vendor/github.com/spf13/cobra/command.go:560 +0x2f4
main.main()
	/var/cache/omnibus/src/registry/src/github.com/docker/distribution/cmd/registry/main.go:23 +0x2d

Failed to run garbage-collect command, starting registry service.
ok: run: registry: (pid 20249) 1s

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to GitLab environment info

root@gitlab:~#  gitlab-rake gitlab:env:info
System information
System:		Ubuntu 18.04
Current User:	git
Using RVM:	no
Ruby Version:	2.6.3p62
Gem Version:	2.7.9
Bundler Version:1.17.3
Rake Version:	12.3.2
Redis Version:	3.2.12
Git Version:	2.21.0
Sidekiq Version:5.2.7
Go Version:	unknown
GitLab information
Version:	12.0.3
Revision:	08a51a9db93
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	10.7
URL:		https://gitlab.noumenadigital.com
HTTP Clone URL:	https://gitlab.noumenadigital.com/some-group/some-project.git
SSH Clone URL:	git@gitlab.noumenadigital.com:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: saml
GitLab Shell
Version:	9.3.0
Repository storage paths:

• default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

root@gitlab:~# sudo gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 9.3.0 ? ... OK (9.3.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
3/1 ... yes
3/2 ... yes
3/3 ... yes
3/4 ... yes
3/5 ... yes
3/7 ... yes
10/8 ... yes
10/9 ... yes
10/10 ... yes
1/11 ... yes
3/12 ... yes
10/13 ... yes
14/14 ... yes
14/16 ... yes
10/17 ... yes
3/20 ... yes
3/21 ... yes
18/22 ... yes
3/23 ... yes
3/24 ... yes
3/25 ... yes
3/26 ... yes
10/27 ... yes
22/28 ... yes
13/29 ... yes
3/30 ... yes
3/31 ... yes
10/32 ... yes
3/33 ... yes
1/34 ... yes
25/35 ... yes
18/36 ... yes
3/37 ... yes
37/38 ... yes
3/40 ... yes
39/41 ... yes
3/42 ... yes
53/43 ... yes
53/44 ... yes
53/45 ... yes
53/46 ... yes
53/47 ... yes
3/50 ... yes
10/51 ... yes
3/52 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.3)
Git version >= 2.21.0 ? ... yes (2.21.0)
Git user has default SSH configuration? ... yes
Active users: ... 29
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes

revert to earlier version of docker registry (2.5.2) until docker fix the issue