Skip to content

Unable to login to registry (11.0) "denied: access forbidden"

Summary

Unable to login to container registry, with or without 2FA, using password or personal access token. All attempts result in "denied: access forbidden"

Hosted gitlab-ce 11.0.0 all-in-one docker image LDAP users and 2FA enabled (Also tried with 2FA disabled) Docker 18.05

Steps to reproduce

docker login gitlab.example.com:4567
# gitlab.rb

registry_nginx['enable'] = true
registry_nginx['redirect_http_to_https'] = false
registry_nginx['listen_https'] = nil
registry_external_url 'https://gitlab.example.com:4567'
registry['enable'] = true

# gitlab-rails/etc/gitlab.yaml (generated)

production:
  ## Container Registry
  registry:
    enabled: true
    host: gitlab.example.com
    port: 4567
    api_url: http://localhost:5000 # internal address to the registry, will be used by GitLab to directly communicate with API
    path: /var/opt/gitlab/gitlab-rails/shared/registry
    key: /var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key
    issuer: omnibus-gitlab-issuer
    
# registry/config.yml (generated)

auth:
  token:
    realm: http://gitlab.example.com/jwt/auth
    service: container_registry
    issuer: omnibus-gitlab-issuer
    rootcertbundle: /var/opt/gitlab/registry/gitlab-registry.crt

What is the current bug behavior?

$ docker login gitlab.example.com:4567
Username: me
Password: 
Error response from daemon: Get https://gitlab.example.com:4567/v2/: denied: access forbidden

What is the expected correct behavior?

$ docker login gitlab.example.com:4567
Username: me
Password: 
Login Succeeded

Relevant logs and/or screenshots

2018-06-24_17:46:07.76145 time="2018-06-24T17:46:07.761403378Z" level=warning msg="error authorizing context: authorization token required" environment=production go.version=go1.9.2 http.request.host="gitlab.example.com:4567" http.request.id=df2243e9-346d-4e2d-90dd-754cfcbe2e8a http.request.method=GET http.request.remoteaddr=192.168.0.138 http.request.uri="/v2/" http.request.useragent="docker/18.05.0-ce go/go1.10.1 git-commit/f150324 kernel/4.13.0/linux arch/amd64 UpstreamClient(Docker-Client/18.05.0-ce \\(linux\\))" instance.id=f03da0d0-aace-46a5-9044-903517b986db service=registry version=v2.6.2-2-g91c17ef 
2018-06-24_17:46:07.76149 127.0.0.1 - - [24/Jun/2018:17:46:07 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/18.05.0-ce go/go1.10.1 git-commit/f150324 kernel/4.13.0/linux arch/amd64 UpstreamClient(Docker-Client/18.05.0-ce \\(linux\\))"

==> /var/log/gitlab/nginx/gitlab_registry_access.log <==
192.168.0.138 - - [24/Jun/2018:17:46:07 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/18.05.0-ce go/go1.10.1 git-commit/f150324 kernel/4.13.0/linux arch/amd64 UpstreamClient(Docker-Client/18.05.0-ce \x5C(linux\x5C))"

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/jwt/auth?account=me&client_id=docker&offline_token=[FILTERED]&service=container_registry" for 172.17.0.1 at 2018-06-24 17:46:07 +0000
Processing by JwtController#auth as HTML
  Parameters: {"account"=>"me", "client_id"=>"docker", "offline_token"=>"[FILTERED]", "service"=>"container_registry"}
Completed 403 Forbidden in 3ms (Views: 0.2ms | ActiveRecord: 0.0ms)

==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/jwt/auth","format":"html","controller":"JwtController","action":"auth","status":403,"duration":3.61,"view":0.17,"db":0.0,"time":"2018-06-24T17:46:07.882Z","params":[{"key":"account","value":"me"},{"key":"client_id","value":"docker"},{"key":"offline_token","value":"[FILTERED]"},{"key":"service","value":"container_registry"}],"remote_ip":"172.17.0.1","user_id":null,"username":null}

==> /var/log/gitlab/gitlab-workhorse/current <==
2018-06-24_17:46:07.88684 gitlab.example.com @ - - [2018/06/24:17:46:07 +0000] "GET /jwt/auth?account=me&client_id=docker&offline_token=true&service=container_registry HTTP/1.1" 403 77 "http://gitlab.example.com/jwt/auth?account=me&client_id=docker&offline_token=true&service=container_registry" "docker/18.05.0-ce go/go1.10.1 git-commit/f150324 kernel/4.13.0/linux arch/amd64 UpstreamClient(Docker-Client/18.05.0-ce \\(linux\\))" 0.012

==> /var/log/gitlab/nginx/gitlab_access.log <==
172.17.0.1 - - [24/Jun/2018:17:46:07 +0000] "GET /jwt/auth?account=me&client_id=docker&offline_token=true&service=container_registry HTTP/1.1" 403 77 "http://gitlab.example.com/jwt/auth" "docker/18.05.0-ce go/go1.10.1 git-commit/f150324 kernel/4.13.0/linux arch/amd64 UpstreamClient(Docker-Client/18.05.0-ce \x5C(linux\x5C))"

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   2.4.4p296
Gem Version:    2.7.6
Bundler Version:1.16.2
Rake Version:   12.3.1
Redis Version:  3.2.11
Git Version:    2.17.1
Sidekiq Version:5.1.3
Go Version:     unknown


GitLab information
Version:        11.0.0
Revision:       b84bfb5
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            http://gitlab.example.com
HTTP Clone URL: http://gitlab.example.com/some-group/some-project.git
SSH Clone URL:  ssh://git@gitlab.example.com:9922/some-group/some-project.git
Using LDAP:     yes
Using Omniauth: no


GitLab Shell
Version:        7.1.4
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab Shell ...


GitLab Shell version >= 7.1.4 ? ... OK (7.1.4)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
2/1 ... ok
2/2 ... ok
2/3 ... ok
2/4 ... ok
5/5 ... ok
5/6 ... ok
5/7 ... ok
5/8 ... ok
2/9 ... ok
2/10 ... ok
2/12 ... ok
2/13 ... ok
2/14 ... ok
2/15 ... ok
2/16 ... repository is empty
2/17 ... ok
2/18 ... ok
2/19 ... ok
2/20 ... ok
2/21 ... ok
2/22 ... ok
2/23 ... ok
2/24 ... ok
2/25 ... ok
2/26 ... ok
2/27 ... ok
2/28 ... ok
2/30 ... ok
2/31 ... ok
2/32 ... ok
2/33 ... ok
2/34 ... ok
2/35 ... ok
2/36 ... ok
2/37 ... ok
2/38 ... repository is empty
2/39 ... repository is empty
2/40 ... ok
2/41 ... ok
2/46 ... ok
2/47 ... ok
2/48 ... ok
9/49 ... ok
2/50 ... repository is empty
4/51 ... ok
2/52 ... ok
2/53 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Sidekiq ...


Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Reply by email is disabled in config/gitlab.yml
Checking LDAP ...


Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
[REDACTED]


Checking LDAP ... Finished


Checking GitLab ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
2/1 ... yes
2/2 ... yes
2/3 ... yes
2/4 ... yes
5/5 ... yes
5/6 ... yes
5/7 ... yes
5/8 ... yes
2/9 ... yes
2/10 ... yes
2/12 ... yes
2/13 ... yes
2/14 ... yes
2/15 ... yes
2/16 ... yes
2/17 ... yes
2/18 ... yes
2/19 ... yes
2/20 ... yes
2/21 ... yes
2/22 ... yes
2/23 ... yes
2/24 ... yes
2/25 ... yes
2/26 ... yes
2/27 ... yes
2/28 ... yes
2/30 ... yes
2/31 ... yes
2/32 ... yes
2/33 ... yes
2/34 ... yes
2/35 ... yes
2/36 ... yes
2/37 ... yes
2/38 ... yes
2/39 ... yes
2/40 ... yes
2/41 ... yes
2/46 ... yes
2/47 ... yes
2/48 ... yes
9/49 ... yes
2/50 ... yes
4/51 ... yes
2/52 ... yes
2/53 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.4.4)
Git version >= 2.9.5 ? ... yes (2.17.1)
Git user has default SSH configuration? ... yes
Active users: ... 5


Checking GitLab ... Finished