docker login always causes "access forbidden"

Summary

Hey guys, for hours now I try to figure out why I can't login to my container registry.

I have absolutely no idea why I can't login :-(

Did I forget something?

Steps to reproduce

docker login git.my.domain:4567
Username: justme
Password:
Error response from daemon: Get https://git.my.domain:4567/v2/: denied: access forbidden

What is the current bug behavior?

When I try to login I get "denied: access forbidden"

My registry docker-compose.yml looks like this:

registry:
  image: registry:2
  environment:
    VIRTUAL_HOST: "reg.my.domain"
    LETSENCRYPT_HOST: "reg.my.domain"
    LETSENCRYPT_EMAIL: "info@my.domain"
    REGISTRY_HTTP_TLS_CERTIFICATE: "/certs/reg.my.domain/fullchain.pem"
    REGISTRY_HTTP_TLS_KEY: "/certs/reg.my.domain/key.pem"
    REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: "/certs/reg.schmid.digital/cert.pem"
    REGISTRY_AUTH_TOKEN_REALM: "https://git.my.domain/jwt/auth"
    REGISTRY_AUTH_TOKEN_SERVICE: "container_registry"
    REGISTRY_AUTH_TOKEN_ISSUER: "gitlab-issuer"
    VIRTUAL_PORT: 5000
  volumes:
    - "./certs:/certs"
  ports:
    - "5000:5000"
  restart: always
  container_name: registry

My gitlab.rb looks like this:

registry_external_url 'https://git.my.domain:4567'

# Settings used by GitLab application
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "git.my.domain"
gitlab_rails['registry_port'] = "4567"
gitlab_rails['registry_api_url'] = "https://reg.my.domain:5000"
gitlab_rails['registry_key_path'] = "/var/opt/gitlab/gitlab-rails/certificate.key"
gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"
gitlab_rails['registry_issuer'] = "gitlab-issuer"

# Settings used by Registry application
registry['enable'] = true

What is the expected correct behavior?

It should say: Login success :)

Relevant logs and/or screenshots

gitlab_1  | 2017-03-02_14:20:55.67891 time="2017-03-02T14:20:55.678870001Z" level=warning msg="error authorizing context: authorization token required" environment=production go.version=go1.5.4 http.request.host="git.my.domain:4567" http.request.id=15e87d67-e4ab-47a9-89e2-200cbf11bb3d http.request.method=GET http.request.remoteaddr=62.152.183.197 http.request.uri="/v2/" http.request.useragent="docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/4.9.8-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(darwin\\))" instance.id=69ccc05d-bcaa-445a-a7bc-750f9f973cb8 service=registry version=v2.4.1
gitlab_1  | 2017-03-02_14:20:55.67893 127.0.0.1 - - [02/Mar/2017:14:20:55 +0000] "GET /v2/ HTTP/1.0" 401 87 "" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/4.9.8-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(darwin\\))"
gitlab_1  |
gitlab_1  | ==> /var/log/gitlab/nginx/gitlab_registry_access.log <==
gitlab_1  | 62.152.183.197 - - [02/Mar/2017:14:20:55 +0000] "GET /v2/ HTTP/1.1" 401 87 "-" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/4.9.8-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \x5C(darwin\x5C))"
gitlab_1  |
gitlab_1  | ==> /var/log/gitlab/gitlab-rails/production.log <==
gitlab_1  | Started GET "/jwt/auth?account=fdsf&client_id=docker&offline_token=true&service=container_registry" for 172.17.0.22 at 2017-03-02 14:20:55 +0000
gitlab_1  | Processing by JwtController#auth as HTML
gitlab_1  |   Parameters: {"account"=>"fdsf", "client_id"=>"docker", "offline_token"=>"true", "service"=>"container_registry"}
gitlab_1  | Completed 403 Forbidden in 10ms (Views: 0.1ms | ActiveRecord: 0.9ms)
gitlab_1  |
gitlab_1  | ==> /var/log/gitlab/gitlab-workhorse/current <==
gitlab_1  | 2017-03-02_14:20:55.81356 git.my.domain @ - - [2017-03-02 14:20:55.796013062 +0000 UTC] "GET /jwt/auth?account=fdsf&client_id=docker&offline_token=true&service=container_registry HTTP/1.1" 403 77 "http://git.my.domain/jwt/auth?account=fdsf&client_id=docker&offline_token=true&service=container_registry" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/4.9.8-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(darwin\\))" 0.017521
gitlab_1  |
gitlab_1  | ==> /var/log/gitlab/nginx/gitlab_access.log <==
gitlab_1  | 172.17.0.22 - - [02/Mar/2017:14:20:55 +0000] "GET /jwt/auth?account=fdsf&client_id=docker&offline_token=true&service=container_registry HTTP/1.1" 403 103 "http://git.my.domain/jwt/auth?account=fdsf&client_id=docker&offline_token=true&service=container_registry" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/4.9.8-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \x5C(darwin\x5C))"

Results of GitLab environment info

System information System: Current User: git Using RVM: no Ruby Version: 2.3.3p222 Gem Version: 2.6.6 Bundler Version:1.13.7 Rake Version: 10.5.0 Redis Version: 3.2.5 Sidekiq Version:4.2.7

GitLab information Version: 8.17.2 Revision: bab14bdb Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: http://git.my.domain HTTP Clone URL: http://git.my.domain/some-group/some-project.git SSH Clone URL: ssh://git@git.my.domain:10022/some-group/some-project.git Using LDAP: no Using Omniauth: no

GitLab Shell Version: 4.1.1 Repository storage paths:

default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks/ Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Checking GitLab Shell ...

GitLab Shell version >= 4.1.1 ? ... OK (4.1.1) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 39/4 ... ok 41/5 ... ok 41/7 ... ok 47/8 ... ok 48/9 ... ok 48/10 ... ok 49/11 ... ok 50/12 ... ok 50/13 ... ok 50/14 ... ok 47/22 ... ok 39/23 ... ok 47/24 ... ok 42/26 ... ok 39/27 ... ok 39/28 ... ok 42/31 ... ok 42/32 ... ok 42/34 ... ok 42/35 ... ok 42/36 ... ok 42/37 ... ok 53/40 ... ok 55/41 ... ok 42/42 ... ok 42/43 ... ok 42/44 ... ok 42/45 ... ok 42/46 ... ok 42/47 ... ok 42/48 ... ok 42/49 ... ok 42/50 ... ok 42/51 ... ok 42/52 ... ok 42/53 ... ok 42/54 ... ok 42/55 ... ok 40/57 ... ok 42/58 ... ok 42/59 ... ok 42/60 ... ok 39/61 ... ok 35/62 ... repository is empty 42/63 ... ok 42/64 ... ok 56/65 ... ok 42/66 ... ok 42/70 ... ok 42/71 ... ok 42/72 ... ok 42/74 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK Send ping to redis server: OK gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Reply by email ...

Reply by email is disabled in config/gitlab.yml

Checking Reply by email ... Finished

Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Git configured with autocrlf=input? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config outdated? ... no Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory setup correctly? ... no Try fixing it: sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} ; sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} ; For more information see: doc/install/installation.md in section "GitLab" Please fix the error above and rerun the checks. Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) projects have namespace: ... 39/4 ... yes 41/5 ... yes 41/7 ... yes 47/8 ... yes 48/9 ... yes 48/10 ... yes 49/11 ... yes 50/12 ... yes 50/13 ... yes 50/14 ... yes 47/22 ... yes 39/23 ... yes 47/24 ... yes 42/26 ... yes 39/27 ... yes 39/28 ... yes 42/31 ... yes 42/32 ... yes 42/34 ... yes 42/35 ... yes 42/36 ... yes 42/37 ... yes 53/40 ... yes 55/41 ... yes 42/42 ... yes 42/43 ... yes 42/44 ... yes 42/45 ... yes 42/46 ... yes 42/47 ... yes 42/48 ... yes 42/49 ... yes 42/50 ... yes 42/51 ... yes 42/52 ... yes 42/53 ... yes 42/54 ... yes 42/55 ... yes 40/57 ... yes 42/58 ... yes 42/59 ... yes 42/60 ... yes 39/61 ... yes 35/62 ... yes 42/63 ... yes 42/64 ... yes 56/65 ... yes 42/66 ... yes 42/70 ... yes 42/71 ... yes 42/72 ... yes 42/74 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.1.0 ? ... yes (2.3.3) Your git bin path is "/opt/gitlab/embedded/bin/git" Git version >= 2.7.3 ? ... yes (2.10.2) Active users: 7

Checking GitLab ... Finished