Pulling from docker/k8s failed with denied: access forbidden

Summary

Suddenly yesterday our CI jobs fail with error: Failed to pull image "registry.domain/tag": rpc error: code = Unknown desc = Error response from daemon: Get https://registry.domain/tag: denied: access forbidden

I tried login with my current username and password and it works without problem, but deploy a release from Gitlab results in docker image pull failed.

I checked imagePullSecrets in k8s config and it's using gitlab-registry with content

{"auths":{"registry.domain":{"username":"gitlab-ci-token","password":"password","email":"email","auth":"..."}}}

I tried login with username and password above and got unauthorized: HTTP Basic: Access denied

Started GET "/jwt/auth?account=gitlab-ci-token&client_id=docker&offline_token=[FILTERED]&service=container_registry" for ip at 2017-12-17 18:37:23 +0700
Processing by JwtController#auth as HTML
  Parameters: {"account"=>"gitlab-ci-token", "client_id"=>"docker", "offline_token"=>"[FILTERED]", "service"=>"container_registry"}
Filter chain halted as :authenticate_project_or_user rendered or redirected
Completed 401 Unauthorized in 44ms (Views: 0.4ms | ActiveRecord: 4.0ms)

It's working just yesterday and I haven't made any changes since or before then.

Relevant logs and/or screenshots

# /var/log/gitlab/gitlab-rails/production.log
Started GET "/jwt/auth?scope=repository%3Atag%3Apull&service=container_registry" for ip at 2017-12-17 18:06:57 +0700
Processing by JwtController#auth as HTML
  Parameters: {"scope"=>"repository:tag:pull", "service"=>"container_registry"}
Completed 403 Forbidden in 16ms (Views: 0.7ms | ActiveRecord: 2.0ms)

# /var/log/caddy/gitlab.log
ip - - [17/Dec/2017:18:11:03 +0700] "GET /jwt/auth?scope=repository%3Atag%3Apull&service=container_registry HTTP/1.1" 403 99

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:         Ubuntu 16.04
Current User:   git
Using RVM:      no
Ruby Version:   2.3.5p376
Gem Version:    2.6.13
Bundler Version:1.13.7
Rake Version:   12.1.0
Redis Version:  3.2.5
Git Version:    2.13.6
Sidekiq Version:5.0.4
Go Version:     go1.6.2 linux/amd64

GitLab information
Version:        10.2.4
Revision:       ba9acca
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            https://domain
HTTP Clone URL: https://domain/some-group/some-project.git
SSH Clone URL:  git@domain:some-group/some-project.git
Using LDAP:     yes
Using Omniauth: no


GitLab Shell
Version:        5.9.4
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab Shell ...

GitLab Shell version >= 5.9.4 ? ... OK (5.9.4)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
4/1 ... ok
6/2 ... ok
15/3 ... ok
6/4 ... ok
6/5 ... ok
6/6 ... ok
6/7 ... ok
8/9 ... ok
6/10 ... ok
8/11 ... ok
15/12 ... ok
6/13 ... ok
6/14 ... ok
7/15 ... ok
9/16 ... ok
10/17 ... ok
11/18 ... ok
12/19 ... ok
8/20 ... ok
12/21 ... ok
7/23 ... ok
12/24 ... ok
10/25 ... ok
9/26 ... ok
6/27 ... ok
7/28 ... ok
10/29 ... ok
7/30 ... ok
9/31 ... ok
10/32 ... ok
12/33 ... ok
7/34 ... ok
7/36 ... ok
7/37 ... ok
4/38 ... ok
7/39 ... ok
11/40 ... ok
9/41 ... ok
12/42 ... ok
10/43 ... ok
10/44 ... ok
12/45 ... ok
6/46 ... repository is empty
17/47 ... ok
4/49 ... ok
17/53 ... ok
7/54 ... ok
4/55 ... ok
6/56 ... ok
17/58 ... ok
7/59 ... ok
7/60 ... ok
9/61 ... ok
8/62 ... ok
6/63 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Sidekiq ...


Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Reply by email is disabled in config/gitlab.yml
Checking LDAP ...


Server: ldapfpt
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
Checking LDAP ... Finished


Checking GitLab ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
4/1 ... yes
6/2 ... yes
15/3 ... yes
6/4 ... yes
6/5 ... yes
6/6 ... yes
6/7 ... yes
8/9 ... yes
6/10 ... yes
8/11 ... yes
15/12 ... yes
6/13 ... yes
6/14 ... yes
7/15 ... yes
9/16 ... yes
10/17 ... yes
11/18 ... yes
12/19 ... yes
8/20 ... yes
12/21 ... yes
7/23 ... yes
12/24 ... yes
10/25 ... yes
9/26 ... yes
6/27 ... yes
7/28 ... yes
10/29 ... yes
7/30 ... yes
9/31 ... yes
10/32 ... yes
12/33 ... yes
7/34 ... yes
7/36 ... yes
7/37 ... yes
4/38 ... yes
7/39 ... yes
11/40 ... yes
9/41 ... yes
12/42 ... yes
10/43 ... yes
10/44 ... yes
12/45 ... yes
6/46 ... yes
17/47 ... yes
4/49 ... yes
17/53 ... yes
7/54 ... yes
4/55 ... yes
6/56 ... yes
17/58 ... yes
7/59 ... yes
7/60 ... yes
9/61 ... yes
8/62 ... yes
6/63 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.3.5)
Git version >= 2.7.3 ? ... yes (2.13.6)
Git user has default SSH configuration? ... yes
Active users: ... 13


Checking GitLab ... Finished
Edited by Công Nguyễn