Unable to upload to the Registry due to permissions issue on _upload directory
Summary
Gitlab Registry encounters a permission denied error when trying to upload a container. We only noticed this bug after upgrading to 9.1.3 but it has been a few weeks since we needed to do this, so it could have been an issue for longer.
Steps to reproduce
Upload a container to the Gitlab registry.
What is the current bug behavior?
Client received a 500 Internal Server Error on upload. Logs indicate a permissions issue when trying to create a directory, see below.
What is the expected correct behavior?
Container uploads successfully.
Relevant logs and/or screenshots
From /var/log/gitlab/registry/current:
2017-05-09_08:04:17.13891 time="2017-05-09T09:04:17.138833779+01:00" level=error msg="response completed with error" auth.user.name=my.user environment=production err.code=unknown err.detail="filesystem: mkdir /var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/repositories/linux-developers/acmeproject/_uploads/d77e1131-21ca-44ff-8154-78821606e309: permission denied" err.message="unknown error" go.version=go1.5.4 http.request.host=registry.acmecorp.com http.request.id=962928dc-6d71-4afb-8c8d-eb3373eeb579 http.request.method=POST http.request.remoteaddr=123.45.67.89 http.request.uri="/v2/linux-developers/acmeproject/blobs/uploads/" http.request.useragent="docker/17.05.0-ce go/go1.7.5 git-commit/89658be kernel/3.10.0-514.16.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.05.0-ce \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.414505ms http.response.status=500 http.response.written=288 instance.id=dca13802-b147-4bf3-ab71-8b652cfa595c service=registry vars.name="linux-developers/acmeproject" version=v2.4.1
2017-05-09_08:04:17.13894 127.0.0.1 - - [09/May/2017:09:04:17 +0100] "POST /v2/linux-developers/acmeproject/blobs/uploads/ HTTP/1.0" 500 288 "" "docker/17.05.0-ce go/go1.7.5 git-commit/89658be kernel/3.10.0-514.16.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.05.0-ce \\(linux\\))"Permissions on the upload directory:
[root@gitlab cdn-ddos]# pwd
/var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/repositories/linux-developers/acmeproject
[root@gitlab acmeproject]# ll
total 12
drwxr-xr-x 3 git git 4096 Jul 19 2016 _layers
drwxr-xr-x 4 git git 4096 Jul 19 2016 _manifests
drwxr-xr-x 2 git git 4096 Mar 15 09:29 _uploadsResults of GitLab environment info
[root@gitlab ~]# sudo gitlab-rake gitlab:env:info
System information System: Current User: git Using RVM: no Ruby Version: 2.3.3p222 Gem Version: 2.6.6 Bundler Version:1.13.7 Rake Version: 10.5.0 Redis Version: 3.2.5 Git Version: 2.11.1 Sidekiq Version:4.2.7
GitLab information Version: 9.1.3 Revision: 2e4e522 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://gitlab.acmecorp.com HTTP Clone URL: https://gitlab.acmecorp.com/some-group/some-project.git SSH Clone URL: git@gitlab.acmecorp.com:some-group/some-project.git Using LDAP: yes Using Omniauth: no
GitLab Shell Version: 5.0.2 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
[root@gitlab ~]# gitlab-rake gitlab:check SANITIZE=true Checking GitLab Shell ...
GitLab Shell version >= 5.0.2 ? ... OK (5.0.2) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ...
... snipped ~600 lines, all ok ...
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK Send ping to redis server: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) ... snipped LDAP user list ....
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config outdated? ... no Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory setup correctly? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) projects have namespace: ...
... snipped ~600 lines, all ok ...
Redis version >= 2.8.0? ... yes Ruby version >= 2.1.0 ? ... yes (2.3.3) Your git bin path is "/opt/gitlab/embedded/bin/git" Git version >= 2.7.3 ? ... yes (2.11.1) Active users: 139
Checking GitLab ... Finished