docker login always causes "access forbidden"
Summary
Hey guys, for hours now I try to figure out why I can't login to my container registry.
I have absolutely no idea why I can't login :-(
Did I forget something?
Steps to reproduce
docker login git.my.domain:4567
Username: justme
Password:
Error response from daemon: Get https://git.my.domain:4567/v2/: denied: access forbidden
What is the current bug behavior?
When I try to login I get "denied: access forbidden"
My registry docker-compose.yml looks like this:
registry:
image: registry:2
environment:
VIRTUAL_HOST: "reg.my.domain"
LETSENCRYPT_HOST: "reg.my.domain"
LETSENCRYPT_EMAIL: "info@my.domain"
REGISTRY_HTTP_TLS_CERTIFICATE: "/certs/reg.my.domain/fullchain.pem"
REGISTRY_HTTP_TLS_KEY: "/certs/reg.my.domain/key.pem"
REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: "/certs/reg.schmid.digital/cert.pem"
REGISTRY_AUTH_TOKEN_REALM: "https://git.my.domain/jwt/auth"
REGISTRY_AUTH_TOKEN_SERVICE: "container_registry"
REGISTRY_AUTH_TOKEN_ISSUER: "gitlab-issuer"
VIRTUAL_PORT: 5000
volumes:
- "./certs:/certs"
ports:
- "5000:5000"
restart: always
container_name: registry
My gitlab.rb looks like this:
registry_external_url 'https://git.my.domain:4567'
# Settings used by GitLab application
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "git.my.domain"
gitlab_rails['registry_port'] = "4567"
gitlab_rails['registry_api_url'] = "https://reg.my.domain:5000"
gitlab_rails['registry_key_path'] = "/var/opt/gitlab/gitlab-rails/certificate.key"
gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"
gitlab_rails['registry_issuer'] = "gitlab-issuer"
# Settings used by Registry application
registry['enable'] = true
What is the expected correct behavior?
It should say: Login success :)
Relevant logs and/or screenshots
gitlab_1 | 2017-03-02_14:20:55.67891 time="2017-03-02T14:20:55.678870001Z" level=warning msg="error authorizing context: authorization token required" environment=production go.version=go1.5.4 http.request.host="git.my.domain:4567" http.request.id=15e87d67-e4ab-47a9-89e2-200cbf11bb3d http.request.method=GET http.request.remoteaddr=62.152.183.197 http.request.uri="/v2/" http.request.useragent="docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/4.9.8-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(darwin\\))" instance.id=69ccc05d-bcaa-445a-a7bc-750f9f973cb8 service=registry version=v2.4.1
gitlab_1 | 2017-03-02_14:20:55.67893 127.0.0.1 - - [02/Mar/2017:14:20:55 +0000] "GET /v2/ HTTP/1.0" 401 87 "" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/4.9.8-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(darwin\\))"
gitlab_1 |
gitlab_1 | ==> /var/log/gitlab/nginx/gitlab_registry_access.log <==
gitlab_1 | 62.152.183.197 - - [02/Mar/2017:14:20:55 +0000] "GET /v2/ HTTP/1.1" 401 87 "-" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/4.9.8-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \x5C(darwin\x5C))"
gitlab_1 |
gitlab_1 | ==> /var/log/gitlab/gitlab-rails/production.log <==
gitlab_1 | Started GET "/jwt/auth?account=fdsf&client_id=docker&offline_token=true&service=container_registry" for 172.17.0.22 at 2017-03-02 14:20:55 +0000
gitlab_1 | Processing by JwtController#auth as HTML
gitlab_1 | Parameters: {"account"=>"fdsf", "client_id"=>"docker", "offline_token"=>"true", "service"=>"container_registry"}
gitlab_1 | Completed 403 Forbidden in 10ms (Views: 0.1ms | ActiveRecord: 0.9ms)
gitlab_1 |
gitlab_1 | ==> /var/log/gitlab/gitlab-workhorse/current <==
gitlab_1 | 2017-03-02_14:20:55.81356 git.my.domain @ - - [2017-03-02 14:20:55.796013062 +0000 UTC] "GET /jwt/auth?account=fdsf&client_id=docker&offline_token=true&service=container_registry HTTP/1.1" 403 77 "http://git.my.domain/jwt/auth?account=fdsf&client_id=docker&offline_token=true&service=container_registry" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/4.9.8-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(darwin\\))" 0.017521
gitlab_1 |
gitlab_1 | ==> /var/log/gitlab/nginx/gitlab_access.log <==
gitlab_1 | 172.17.0.22 - - [02/Mar/2017:14:20:55 +0000] "GET /jwt/auth?account=fdsf&client_id=docker&offline_token=true&service=container_registry HTTP/1.1" 403 103 "http://git.my.domain/jwt/auth?account=fdsf&client_id=docker&offline_token=true&service=container_registry" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/4.9.8-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \x5C(darwin\x5C))"
Results of GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.3.3p222 Gem Version: 2.6.6 Bundler Version:1.13.7 Rake Version: 10.5.0 Redis Version: 3.2.5 Sidekiq Version:4.2.7
GitLab information Version: 8.17.2 Revision: bab14bdb Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: http://git.my.domain HTTP Clone URL: http://git.my.domain/some-group/some-project.git SSH Clone URL: ssh://git@git.my.domain:10022/some-group/some-project.git Using LDAP: no Using Omniauth: no
GitLab Shell Version: 4.1.1 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks/ Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Checking GitLab Shell ...
GitLab Shell version >= 4.1.1 ? ... OK (4.1.1) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 39/4 ... ok 41/5 ... ok 41/7 ... ok 47/8 ... ok 48/9 ... ok 48/10 ... ok 49/11 ... ok 50/12 ... ok 50/13 ... ok 50/14 ... ok 47/22 ... ok 39/23 ... ok 47/24 ... ok 42/26 ... ok 39/27 ... ok 39/28 ... ok 42/31 ... ok 42/32 ... ok 42/34 ... ok 42/35 ... ok 42/36 ... ok 42/37 ... ok 53/40 ... ok 55/41 ... ok 42/42 ... ok 42/43 ... ok 42/44 ... ok 42/45 ... ok 42/46 ... ok 42/47 ... ok 42/48 ... ok 42/49 ... ok 42/50 ... ok 42/51 ... ok 42/52 ... ok 42/53 ... ok 42/54 ... ok 42/55 ... ok 40/57 ... ok 42/58 ... ok 42/59 ... ok 42/60 ... ok 39/61 ... ok 35/62 ... repository is empty 42/63 ... ok 42/64 ... ok 56/65 ... ok 42/66 ... ok 42/70 ... ok 42/71 ... ok 42/72 ... ok 42/74 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK Send ping to redis server: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config outdated? ... no Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory setup correctly? ... no Try fixing it: sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} ; sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} ; For more information see: doc/install/installation.md in section "GitLab" Please fix the error above and rerun the checks. Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) projects have namespace: ... 39/4 ... yes 41/5 ... yes 41/7 ... yes 47/8 ... yes 48/9 ... yes 48/10 ... yes 49/11 ... yes 50/12 ... yes 50/13 ... yes 50/14 ... yes 47/22 ... yes 39/23 ... yes 47/24 ... yes 42/26 ... yes 39/27 ... yes 39/28 ... yes 42/31 ... yes 42/32 ... yes 42/34 ... yes 42/35 ... yes 42/36 ... yes 42/37 ... yes 53/40 ... yes 55/41 ... yes 42/42 ... yes 42/43 ... yes 42/44 ... yes 42/45 ... yes 42/46 ... yes 42/47 ... yes 42/48 ... yes 42/49 ... yes 42/50 ... yes 42/51 ... yes 42/52 ... yes 42/53 ... yes 42/54 ... yes 42/55 ... yes 40/57 ... yes 42/58 ... yes 42/59 ... yes 42/60 ... yes 39/61 ... yes 35/62 ... yes 42/63 ... yes 42/64 ... yes 56/65 ... yes 42/66 ... yes 42/70 ... yes 42/71 ... yes 42/72 ... yes 42/74 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.1.0 ? ... yes (2.3.3) Your git bin path is "/opt/gitlab/embedded/bin/git" Git version >= 2.7.3 ? ... yes (2.10.2) Active users: 7
Checking GitLab ... Finished