Restoring Container Registry from Backup Results in 500 Error when pushing.
Summary
After moving GitLab to a new server with restoring a backup, pushes to GitLab Container Registry result to a 500 Error.
Steps to reproduce
- Use GitLab Registry on an old server
- Buy and move to a new server and use a backup of the old server.
- Create a new Project with GitLab Registry enabled and try to push.
Expected behavior
Push will success.
Actual behavior
Docker is trying to push to the Registry a couple of times, till resulting in an 500 Error.
Relevant logs and/or screenshots
nginx error log:
2016/09/16 02:26:53 [error] 8629#0: *132 connect() failed (111: Connection refused) while connecting to upstream, client: 178.203.234.71, server: ***, request: "POST /v2/docker/pandoc/blobs/uploads/ HTTP/1.1", upstream: "http://[::1]:5000/v2/docker/pandoc/blobs/uploads/", host: "***:4567"
2016/09/16 02:27:13 [error] 8629#0: *161 connect() failed (111: Connection refused) while connecting to upstream, client: 178.203.234.71, server: ***, request: "POST /v2/docker/pandoc/blobs/uploads/ HTTP/1.1", upstream: "http://[::1]:5000/v2/docker/pandoc/blobs/uploads/", host: "***:4567"registry error log:
/shared/registry/docker/registry/v2/repositories/docker/pandoc: permission denied" err.message="unknown error" go.version=go1.5.4 http.request.host="***:4567" http.request.id=4e9bd8a2-2503-4b46-98c1-0786ace2c193 http.request.method=POST http.request.remoteaddr=178.203.234.71 http.request.uri="/v2/docker/pandoc/blobs/uploads/" http.request.useragent="docker/1.12.0 go/go1.6.3 git-commit/8eab29e kernel/4.4.15-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.0 \\(windows\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=12.390291ms http.response.status=500 http.response.written=230 instance.id=2b689b29-5142-4e53-a7a5-a7be8f5495cc service=registry vars.name="docker/pandoc" version=v2.4.1
2016-09-16_01:03:37.49932 127.0.0.1 - - [16/Sep/2016:03:03:37 +0200] "POST /v2/docker/pandoc/blobs/uploads/ HTTP/1.0" 500 230 "" "docker/1.12.0 go/go1.6.3 git-commit/8eab29e kernel/4.4.15-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.0 \\(windows\\))"
2016-09-16_01:03:37.50192 time="2016-09-16T03:03:37.501691177+02:00" level=error msg="response completed with error" auth.user.name=*** environment=production err.code=unknown err.detail="filesystem: mkdir /var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/repositories/docker/pandoc: permission denied" err.message="unknown error" go.version=go1.5.4 http.request.host="***:4567" http.request.id=910a6020-d198-4f97-9450-6394e73eff0e http.request.method=POST http.request.remoteaddr=178.203.234.71 http.request.uri="/v2/docker/pandoc/blobs/uploads/" http.request.useragent="docker/1.12.0 go/go1.6.3 git-commit/8eab29e kernel/4.4.15-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.0 \\(windows\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=14.64206ms http.response.status=500 http.response.written=230 instance.id=2b689b29-5142-4e53-a7a5-a7be8f5495cc service=registry vars.name="docker/pandoc" version=v2.4.1
2016-09-16_01:03:37.50197 127.0.0.1 - - [16/Sep/2016:03:03:37 +0200] "POST /v2/docker/pandoc/blobs/uploads/ HTTP/1.0" 500 230 "" "docker/1.12.0 go/go1.6.3 git-commit/8eab29e kernel/4.4.15-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.0 \\(windows\\))"
2016-09-16_01:03:37.50239 time="2016-09-16T03:03:37.502283436+02:00" level=error msg="response completed with error" auth.user.name=***environment=production err.code=unknown err.detail="filesystem: mkdir /var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/repositories/docker/pandoc: permission denied" err.message="unknown error" go.version=go1.5.4 http.request.host="***:4567" http.request.id=77b5f782-8997-4569-84c8-a7d9111a8106 http.request.method=POST http.request.remoteaddr=178.203.234.71 http.request.uri="/v2/docker/pandoc/blobs/uploads/" http.request.useragent="docker/1.12.0 go/go1.6.3 git-commit/8eab29e kernel/4.4.15-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.0 \\(windows\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=10.419535ms http.response.status=500 http.response.written=230 instance.id=2b689b29-5142-4e53-a7a5-a7be8f5495cc service=registry vars.name="docker/pandoc" version=v2.4.1
2016-09-16_01:03:37.50246 127.0.0.1 - - [16/Sep/2016:03:03:37 +0200] "POST /v2/docker/pandoc/blobs/uploads/ HTTP/1.0" 500 230 "" "docker/1.12.0 go/go1.6.3 git-commit/8eab29e kernel/4.4.15-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.0 \\(windows\\))"Directorys owners:
New Server:
# ll /var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/repositories/docker/
total 24
drwxr-xr-x 6 git git 4096 Sep 4 04:15 ./
drwxr-xr-x 4 git git 4096 Sep 3 18:55 ../
drwxr-xr-x 5 git git 4096 Sep 3 18:55 nginx/
drwxr-xr-x 5 git git 4096 Sep 3 18:55 php/
drwxr-xr-x 5 git git 4096 Sep 4 06:02 rpi-mono/
drwxr-xr-x 5 git git 4096 Sep 3 18:56 wordpress-data/Old Server:
# ll /var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/repositories/docker/
total 24
drwxr-xr-x 6 registry registry 4096 Sep 4 04:15 ./
drwxr-xr-x 4 registry registry 4096 Sep 3 18:55 ../
drwxr-xr-x 5 registry registry 4096 Sep 3 18:55 nginx/
drwxr-xr-x 5 registry registry 4096 Sep 3 18:55 php/
drwxr-xr-x 5 registry registry 4096 Sep 4 06:02 rpi-mono/
drwxr-xr-x 5 registry registry 4096 Sep 3 18:56 wordpress-data/
Output of checks
Results of GitLab application Check
Checking GitLab Shell ...
GitLab Shell version >= 3.4.0 ? ... OK (3.4.0)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
3/1 ... ok
3/2 ... ok
3/3 ... ok
3/4 ... ok
4/5 ... ok
4/6 ... repository is empty
7/7 ... ok
8/8 ... ok
10/9 ... ok
6/12 ... ok
9/14 ... ok
1/15 ... ok
9/16 ... ok
12/17 ... ok
9/18 ... ok
8/19 ... repository is empty
8/20 ... ok
10/21 ... repository is empty
10/22 ... ok
3/26 ... ok
3/27 ... ok
3/28 ... ok
3/29 ... ok
3/30 ... ok
12/31 ... ok
13/32 ... ok
13/33 ... repository is empty
13/34 ... ok
12/41 ... ok
1/42 ... ok
8/43 ... repository is empty
12/44 ... ok
8/46 ... ok
15/47 ... ok
15/48 ... repository is empty
15/49 ... repository is empty
15/50 ... repository is empty
15/51 ... repository is empty
15/53 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
IMAP server credentials are correct? ... yes
Init.d configured correctly? ... skipped (omnibus-gitlab has no init script)
MailRoom running? ... can't check because of previous errors
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
3/1 ... yes
3/2 ... yes
3/3 ... yes
3/4 ... yes
4/5 ... yes
4/6 ... yes
7/7 ... yes
8/8 ... yes
10/9 ... yes
6/12 ... yes
9/14 ... yes
1/15 ... yes
9/16 ... yes
12/17 ... yes
9/18 ... yes
8/19 ... yes
8/20 ... yes
10/21 ... yes
10/22 ... yes
3/26 ... yes
3/27 ... yes
3/28 ... yes
3/29 ... yes
3/30 ... yes
12/31 ... yes
13/32 ... yes
13/33 ... yes
13/34 ... yes
12/41 ... yes
1/42 ... yes
8/43 ... yes
12/44 ... yes
8/46 ... yes
15/47 ... yes
15/48 ... yes
15/49 ... yes
15/50 ... yes
15/51 ... yes
15/53 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.1)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.7.4)
Active users: 5
Checking GitLab ... FinishedResults of GitLab environment info
System information
System: Ubuntu 16.04
Current User: git
Using RVM: no
Ruby Version: 2.3.1p112
Gem Version: 2.6.6
Bundler Version:2.3.0
Rake Version: 10.5.0
Sidekiq Version:4.1.4
GitLab information
Version: 8.11.6
Revision: 410d751
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL: https://***
HTTP Clone URL: https://***/some-group/some-project.git
SSH Clone URL: git@***:some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers: github
GitLab Shell
Version: 3.4.0
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git: /opt/gitlab/embedded/bin/gitPossible fixes
I think the Backup Restore Process should chown the registry directory to the registry user. This is how I fixed the problem for myself, too.