"permission denied" on registry after restore a 8.10.2 backup
The problem
In this weekend I have migrated a GitLab 8.10.2 from one server to another. After made de backup in the older server, and restore in the new one, start happening some build errors in my pipeline (summarized to maintain clarity):
[...]
Step 16 : CMD foreman start
---> Running in 0f47946da0fb
---> 648b485a8467
Removing intermediate container 0f47946da0fb
Successfully built 648b485a8467
$ docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CONTAINER_LOGIN
Login Succeeded
$ docker push $CONTAINER_TEST_IMAGE
The push refers to a repository [registry.domain/group/project]
c2586bef0f50: Preparing
e469a5bb3e2d: Preparing
559640485b7d: Preparing
[...]
e469a5bb3e2d: Retrying in 3 seconds
c2586bef0f50: Retrying in 3 seconds
e469a5bb3e2d: Retrying in 2 seconds
c2586bef0f50: Retrying in 2 seconds
e469a5bb3e2d: Retrying in 1 second
c2586bef0f50: Retrying in 1 second
received unexpected HTTP status: 500 Internal Server Error
ERROR: Build failed: exit status 1
I have found this info on /var/log/gitlab/registry/current
:
2016-08-01_12:56:21.39346 time="2016-08-01T08:56:21.393337709-04:00" level=error msg="response completed with error" auth.user.name= environment=production err.code=unknown err.detail="filesystem: mkdir /var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/repositories/group/project/_uploads/6c822080-a222-4efa-93f0-1f5d7ab07b34: permission denied" err.message="unknown error" go.version=go1.5.4 http.request.host=registry.domain.tld http.request.id=61fbb708-97d3-4e2b-979e-367ca8b97c5c http.request.method=POST http.request.remoteaddr=104.131.108.41 http.request.uri="/v2/group/project/blobs/uploads/" http.request.useragent="docker/1.11.2 go/go1.5.4 git-commit/b9f10c9 kernel/3.13.0-85-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.2 \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=10.844198ms http.response.status=500 http.response.written=292 instance.id=3ac0590f-08b0-49a1-9727-5123d11705f2 service=registry vars.name="group/project" version=v2.4.1
And this when run gitlab-ctl tail
:
ip_address - - [01/Aug/2016:08:55:31 -0400] "POST /v2/group/project/blobs/uploads/ HTTP/1.1" 500 292 "-" "docker/1.11.2 go/go1.5.4 git-commit/b9f10c9 kernel/3.13.0-85-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.2 \x5C(linux\x5C))"
To solve this problem, I have compared the old installation and the new one, and found one difference in the /var/opt/gitlab/gitlab-rails/shared/registry
directory: the docker
directory in the old one has the registry:registry
owner/group, and the new one was made with the git:git
owner/group. The same to the subdirectories.
The solution
I fix the problem running this in the new server:
cd /var/opt/gitlab/gitlab-rails/shared/registry
chown -R registry:registry docker/
After this, I re-run my failed builds, and all work again.