[Container Registry] Dock push failed due to unauthorized: authentication required after restoring from backup file

Summary

We failed to use the container registry after restoring from our old backups file.

We are running a GitLab image with an external Nginx as its proxy. The backup file is packed under the version gitlab/gitlab-ce:11.7.7-ce.0 and we upgraded to version gitlab/gitlab-ce:12.10.14-ce.0 after restoring.

The docker login works fine, yet the docker push will be blocked by unauthorized: authentication required, while the tags, however, will be pushed to our GitLab and stored in /var/opt/gitlab/gitlab-rails/shared/registry successfully.

root@gitlab:/var/opt/gitlab/gitlab-rails/shared/registry# tree .
.
└── docker
    └── registry
        └── v2
            └── repositories
                ├── alberthsies
                │   └── containertesting
                │       ├── _uploads
                │       │   ├── <upload-hash>
                │       │   │   ├── data
                │       │   │   ├── hashstates
                │       │   │   │   └── sha256
                │       │   │   │       └── 0
                │       │   │   └── startedat
                │       │   ├── <upload-hash>
                │       │   │   ├── data
                │       │   │   ├── hashstates
                │       │   │   │   └── sha256
                │       │   │   │       └── 0
                │       │   │   └── startedat

Screen_Shot_2020-07-09_at_10.45.59_AM

According to the document, unauthorized: authentication required might occur when the image itself is too large that the token duration timeout, yet, as you can see, the image I tested is really small, and I already adjust the duration to 15 mins. Thus, I believe the duration is not the reason for causing this.

Steps to reproduce

docker-compose.yml 

version: "3.6"
services:
  gitlab:
    image: gitlab/gitlab-ce:12.10.14-ce.0
    restart: always
    hostname: 'my-gitlab.com'
    privileged: true
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'http://my-gitlab.com:80'
        registry_external_url 'http://my-gitlab-registry.com'
        registry['enable'] = true
        registry_nginx['enable'] = true
        registry_nginx['listen_port'] = 5050
        gitlab_rails['registry_enabled'] = true
        gitlab_rails['backup_keep_time'] = 2400
    ports:
      - '8081:80'
      - '443:443'
      - '5050:5050'
    volumes:
      - '/var/gitlab-staging/data:/var/opt/gitlab'
      - '/var/gitlab-staging/logs:/var/log/gitlab'
      - '/var/gitlab-staging/config:/etc/gitlab'
      - '/var/gitlab-staging/backups:/var/opt/gitlab/backups'

external Nginx 

server {
        listen 443 ssl;
        server_name my-gitlab.com;
        ssl_certificate /etc/nginx/ssl/ssl-cert-chained.pem;
        ssl_certificate_key /etc/nginx/ssl/ssl-cert.key;
        location / {
                proxy_pass http://gitlab-ip:8081;
                proxy_redirect off;
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header X-Forwarded-Protocol $scheme;
                proxy_set_header X-Url-Scheme $scheme;
        }
}

server {
        listen 443 ssl;
        listen 80;
        server_name my-gitlab-registry.com;
        ssl_certificate /etc/nginx/ssl/ssl-cert-canvas-chained.pem;
        ssl_certificate_key /etc/nginx/ssl/ssl-cert-canvas.key;
        location / {
                proxy_pass http://gitlab-ip:5050;
                proxy_redirect off;
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header X-Forwarded-Protocol $scheme;
                proxy_set_header X-Url-Scheme $scheme;
        }
}

What is the current bug behavior?

The docker login works fine with both password and token.

root@testing:~/workspace/containertesting$ docker login my-gitlab-registry.com -u alberthsies -p <my_token>
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /home/alberthsies/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

The tag was built as the Gitlab page's command

root@testing:~/workspace/containertesting$ docker build -t my-gitlab-registry.com/alberthsies/containertesting .
Sending build context to Docker daemon  110.6kB
Step 1/3 : FROM python
 ---> <hash>
Step 2/3 : ADD test.py /
 ---> Using cache
 ---> <hash>
Step 3/3 : CMD python test.py
 ---> Using cache
 ---> <hash>
Successfully built <hash>
Successfully tagged my-gitlab-registry.com/alberthsies/containertesting:latest

However, the docker push was block by the unauthorized: authentication required as follow

root@testing:~/workspace/containertesting$ docker push my-gitlab-registry.com/alberthsies/containertesting
The push refers to repository [my-gitlab-registry.com/alberthsies/containertesting ]
<hash>: Pushing [==================================================>]   15.36kB
unauthorized: authentication required

What is the expected correct behavior?

docker push success, and the images appear on our GitLab.

Relevant logs and/or screenshots

The logs seem fine too.

Logs for docker login

/var/log/gitlab/registry/current 
2020-07-09_03:30:27.47844 time="2020-07-09T03:30:27.478355463Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.13.9 http.request.host=my-gitlab-registry.com http.request.id=eb4c0980-62b8-4941-bf5d-8df37c7d677a http.request.method=GET http.request.remoteaddr=<Nginx-ip> http.request.uri="/v2/" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b7f0 kernel/4.4.0-184-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(linux\))"
2020-07-09_03:30:27.47850 127.0.0.1 - - [09/Jul/2020:03:30:27 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b7f0 kernel/4.4.0-184-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(linux\\))"
2020-07-09_03:30:27.71868 time="2020-07-09T03:30:27.718606084Z" level=info msg="authorized request" auth.user.name=alberthsies go.version=go1.13.9 http.request.host=my-gitlab-registry.com http.request.id=0070878c-3715-4df6-8fa2-c1a5c99c6356 http.request.method=GET http.request.remoteaddr=<Nginx-ip> http.request.uri="/v2/" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b7f0 kernel/4.4.0-184-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(linux\))"
2020-07-09_03:30:27.71874 time="2020-07-09T03:30:27.718704214Z" level=info msg="response completed" go.version=go1.13.9 http.request.host=my-gitlab-registry.com http.request.id=0070878c-3715-4df6-8fa2-c1a5c99c6356 http.request.method=GET http.request.remoteaddr=<Nginx-ip> http.request.uri="/v2/" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b7f0 kernel/4.4.0-184-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(linux\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=3.684929ms http.response.status=200 http.response.written=2
2020-07-09_03:30:27.71876 127.0.0.1 - - [09/Jul/2020:03:30:27 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b7f0 kernel/4.4.0-184-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(linux\\))"
/var/log/gitlab/gitlab-rails/production.log 
Started GET "/jwt/auth?account=alberthsies&client_id=docker&offline_token=[FILTERED]&service=container_registry" for <Nginx-ip> at 2020-07-09 03:34:44 +0000
Processing by JwtController#auth as HTML
  Parameters: {"account"=>"alberthsies", "client_id"=>"docker", "offline_token"=>"[FILTERED]", "service"=>"container_registry"}
Completed 200 OK in 130ms (Views: 0.4ms | ActiveRecord: 71.0ms | Elasticsearch: 0.0ms | Allocations: 13648)
Started POST "/api/v4/jobs/request" for <Nginx-ip> at 2020-07-09 03:35:03 +0000
Started GET "/help" for 127.0.0.1 at 2020-07-09 03:35:03 +0000
Processing by HelpController#index as */*
Completed 200 OK in 447ms (Views: 433.6ms | ActiveRecord: 3.8ms | Elasticsearch: 0.0ms | Allocations: 338746)

Logs for docker push

/var/log/gitlab/registry/current 
2020-07-09_03:29:13.58590 time="2020-07-09T03:29:13.585800936Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.13.9 http.request.host=my-gitlab-registry.com/ http.request.id=0d2f6066-89e8-41d0-a9b0-b41ac398966a http.request.method=GET http.request.remoteaddr=<Nginx-ip> http.request.uri="/v2/" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b7f0 kernel/4.4.0-184-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(linux\))"
2020-07-09_03:29:13.58592 127.0.0.1 - - [09/Jul/2020:03:29:13 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b7f0 kernel/4.4.0-184-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(linux\\))"
2020-07-09_03:29:13.79558 time="2020-07-09T03:29:13.795488539Z" level=info msg="authorized request" auth.user.name=alberthsies go.version=go1.13.9 http.request.host=my-gitlab-registry.com http.request.id=3e632682-bba3-4f3c-b537-026376d94851 http.request.method=HEAD http.request.remoteaddr=<Nginx-ip> http.request.uri="/v2/alberthsies/containertesting/blobs/sha256:0e03bdcc26d7a9a57ef3b6f1bf1a210cff6239bff7c8cac72435984032851689" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b7f0 kernel/4.4.0-184-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(linux\))" vars.digest="sha256:0e03bdcc26d7a9a57ef3b6f1bf1a210cff6239bff7c8cac72435984032851689" vars.name="alberthsies/containertesting"
2020-07-09_03:29:13.80540 time="2020-07-09T03:29:13.805308597Z" level=error msg="response completed with error" auth.user.name=alberthsies err.code="blob unknown" err.detail=sha256:0e03bdcc26d7a9a57ef3b6f1bf1a210cff6239bff7c8cac72435984032851689 err.message="blob unknown to registry" go.version=go1.13.9 http.request.host=my-gitlab-registry.com http.request.id=3e632682-bba3-4f3c-b537-026376d94851 http.request.method=HEAD http.request.remoteaddr=<Nginx-ip> http.request.uri="/v2/alberthsies/containertesting/blobs/sha256:0e03bdcc26d7a9a57ef3b6f1bf1a210cff6239bff7c8cac72435984032851689" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b7f0 kernel/4.4.0-184-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(linux\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=12.810344ms http.response.status=404 http.response.written=157 vars.digest="sha256:0e03bdcc26d7a9a57ef3b6f1bf1a210cff6239bff7c8cac72435984032851689" vars.name="alberthsies/containertesting"
2020-07-09_03:29:13.80546 127.0.0.1 - - [09/Jul/2020:03:29:13 +0000] "HEAD /v2/alberthsies/containertesting/blobs/sha256:0e03bdcc26d7a9a57ef3b6f1bf1a210cff6239bff7c8cac72435984032851689 HTTP/1.1" 404 157 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b7f0 kernel/4.4.0-184-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(linux\\))"
2020-07-09_03:29:13.82094 time="2020-07-09T03:29:13.820851692Z" level=info msg="authorized request" auth.user.name=alberthsies go.version=go1.13.9 http.request.host=my-gitlab-registry.com http.request.id=6989b64c-a842-41a7-a81d-c3b55f10fe62 http.request.method=POST http.request.remoteaddr=<Nginx-ip> http.request.uri="/v2/alberthsies/containertesting/blobs/uploads/" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b7f0 kernel/4.4.0-184-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(linux\))" vars.name="alberthsies/containertesting"
2020-07-09_03:29:13.82718 time="2020-07-09T03:29:13.827096065Z" level=info msg="response completed" go.version=go1.13.9 http.request.host=my-gitlab-registry.com http.request.id=6989b64c-a842-41a7-a81d-c3b55f10fe62 http.request.method=POST http.request.remoteaddr=<Nginx-ip> http.request.uri="/v2/alberthsies/containertesting/blobs/uploads/" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b7f0 kernel/4.4.0-184-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(linux\))" http.response.duration=10.606996ms http.response.status=202 http.response.written=0
2020-07-09_03:29:13.82719 127.0.0.1 - - [09/Jul/2020:03:29:13 +0000] "POST /v2/alberthsies/containertesting/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b7f0 kernel/4.4.0-184-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(linux\\))"
2020-07-09_03:29:13.83694 time="2020-07-09T03:29:13.836893744Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.13.9 http.request.host=my-gitlab-registry.com http.request.id=f013e50f-0426-4552-b3fd-6ff0cede5e48 http.request.method=PATCH http.request.remoteaddr=<Nginx-ip> http.request.uri="/v2/alberthsies/containertesting/blobs/uploads/c792e589-d398-4647-9d05-7197fdba9c70?_state=Uh_fgH_md0prsfmSLvjTnAVHnNonhx12Cgd878R2gMx7Ik5hbWUiOiJhbGJlcnRoc2llcy9jb250YWluZXJ0ZXN0aW5nIiwiVVVJRCI6ImM3OTJlNTg5LWQzOTgtNDY0Ny05ZDA1LTcxOTdmZGJhOWM3MCIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMC0wNy0wOVQwMzoyOToxMy44MjEwMDMyMjNaIn0%3D" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b7f0 kernel/4.4.0-184-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(linux\))" vars.name="alberthsies/containertesting" vars.uuid=c792e589-d398-4647-9d05-7197fdba9c70
2020-07-09_03:29:13.83697 127.0.0.1 - - [09/Jul/2020:03:29:13 +0000] "PATCH /v2/alberthsies/containertesting/blobs/uploads/c792e589-d398-4647-9d05-7197fdba9c70?_state=Uh_fgH_md0prsfmSLvjTnAVHnNonhx12Cgd878R2gMx7Ik5hbWUiOiJhbGJlcnRoc2llcy9jb250YWluZXJ0ZXN0aW5nIiwiVVVJRCI6ImM3OTJlNTg5LWQzOTgtNDY0Ny05ZDA1LTcxOTdmZGJhOWM3MCIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMC0wNy0wOVQwMzoyOToxMy44MjEwMDMyMjNaIn0%3D HTTP/1.1" 401 258 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b7f0 kernel/4.4.0-184-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(linux\\))"
/var/log/gitlab/gitlab-rails/production.log 
Started GET "/jwt/auth?account=alberthsies&scope=repository%3Aalberthsies%2Fcontainertesting%3Apush%2Cpull&service=container_registry" for <Nginx-ip> at 2020-07-09 03:35:48 +0000
Processing by JwtController#auth as HTML
  Parameters: {"account"=>"alberthsies", "scope"=>"repository:alberthsies/containertesting:push,pull", "service"=>"container_registry"}
Completed 200 OK in 162ms (Views: 0.5ms | ActiveRecord: 79.3ms | Elasticsearch: 0.0ms | Allocations: 17761)
Started POST "/api/v4/jobs/request" for <Nginx-ip> at 2020-07-09 03:36:03 +0000
Started GET "/help" for 127.0.0.1 at 2020-07-09 03:36:04 +0000
Processing by HelpController#index as */*
Completed 200 OK in 407ms (Views: 396.0ms | ActiveRecord: 4.6ms | Elasticsearch: 0.0ms | Allocations: 339059)

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to GitLab environment info 

gitlab-rake gitlab:env:info

System information
System:
Current User:	git
Using RVM:	no
Ruby Version:	2.6.6p146
Gem Version:	2.7.10
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	5.0.7
Git Version:	2.26.2
Sidekiq Version:5.2.7
Go Version:	unknown

GitLab information
Version:	12.10.14
Revision:	fe3e5d62b3e
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	11.7
URL:		http://my-gitlab.com
HTTP Clone URL:	http://my-gitlab.com/some-group/some-project.git
SSH Clone URL:	git@my-gitlab.com:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers:

GitLab Shell
Version:	12.2.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
gitlab-rake gitlab:check SANITIZE=true


Checking GitLab subtasks ...


Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 12.2.0 ? ... OK (12.2.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
1/1 ... yes
5/2 ... yes
8/3 ... yes
5/4 ... yes
8/6 ... yes
8/8 ... yes
8/9 ... yes
8/10 ... yes
7/14 ... yes
17/15 ... yes
17/16 ... yes
6/18 ... yes
17/20 ... yes
17/21 ... yes
6/22 ... yes
8/23 ... yes
8/24 ... yes
17/25 ... yes
17/26 ... yes
17/27 ... yes
6/28 ... yes
17/31 ... yes
8/33 ... yes
6/34 ... yes
6/35 ... yes
21/36 ... yes
21/37 ... yes
17/38 ... yes
8/39 ... yes
6/40 ... yes
21/42 ... yes
19/43 ... yes
8/44 ... yes
5/45 ... yes
21/46 ... yes
21/47 ... yes
21/48 ... yes
8/49 ... yes
21/50 ... yes
21/52 ... yes
21/53 ... yes
26/54 ... yes
25/55 ... yes
17/56 ... yes
5/57 ... yes
25/58 ... yes
21/59 ... yes
21/60 ... yes
21/61 ... yes
21/62 ... yes
22/63 ... yes
7/64 ... yes
18/66 ... yes
29/67 ... yes
18/68 ... yes
37/69 ... yes
33/70 ... yes
33/75 ... yes
35/76 ... yes
41/78 ... yes
21/81 ... yes
21/82 ... yes
21/83 ... yes
26/84 ... yes
46/85 ... yes
21/86 ... yes
21/87 ... yes
46/88 ... yes
6/89 ... yes
21/90 ... yes
21/91 ... yes
21/93 ... yes
21/94 ... yes
21/97 ... yes
21/99 ... yes
46/100 ... yes
21/101 ... yes
21/102 ... yes
21/103 ... yes
48/105 ... yes
46/106 ... yes
21/107 ... yes
21/110 ... yes
21/111 ... yes
21/112 ... yes
50/113 ... yes
21/114 ... yes
21/115 ... yes
21/116 ... yes
21/117 ... yes
41/118 ... yes
21/119 ... yes
52/120 ... yes
21/121 ... yes
21/122 ... yes
41/123 ... yes
21/124 ... yes
21/125 ... yes
21/126 ... yes
21/127 ... yes
55/128 ... yes
49/129 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.6)
Git version >= 2.22.0 ? ... yes (2.26.2)
Git user has default SSH configuration? ... yes
Active users: ... 26
Is authorized keys file accessible? ... yes


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)