Skip to content

Can't push to gitlab registry, even though authentication via docker login was successful

Summary

After enabling private Gitlab registry (Omnibus installation), sorting out self signed certs problem (my registry is under different domain than gitlab itself), I'm able to login via docker login, but no luck with docker push command. This happens in Gitlab CI (auth via gitlab-ci-token) and in my dev environment (auth via gitlab credentials).

Steps to reproduce

My gitlab.rb

git_data_dir "/var/opt/gitlab/git-data"
external_url 'https://gitlab.jakubigla.net'

gitlab_rails['time_zone'] = 'UTC'
gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['gitlab_email_from'] = "gitlab@gitlab.jakubigla.net"
gitlab_rails['gitlab_email_display_name'] = "Gitlab"
gitlab_rails['gitlab_email_reply_to'] = "gitlab@gitlab.jakubigla.net"
gitlab_rails['gitlab_default_projects_features_container_registry'] = true

registry_external_url "https://registry.jakubigla.net"

nginx['redirect_http_to_https'] = true
nginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab.jakubigla.net.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab.jakubigla.net.key"

registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/registry.jakubigla.net.crt"
registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/registry.jakubigla.net.key"

Sample .gitlab-ci.yml

variables:
  DOCKER_REGISTRY: "registry.jakubigla.net"
  NAMESPACE: "test"
  NAME: "test"
  
build:
  image: docker:latest
  stage: build
  before_script:
    - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $DOCKER_REGISTRY
  script:
    - echo "FROM alpine" > Dockerfile
    - docker build -t "$DOCKER_REGISTRY/$NAMESPACE/$NAME" .
    - docker push "$DOCKER_REGISTRY/$NAMESPACE/$NAME"
  tags:
    - docker
  only:
    - master

Expected behavior

I expect that my freshly built image, can be pushed to my registry with no issues whatsoever, after a successful authentication.

Actual behavior

docker push command throws an error:

unauthorized: authentication required
ERROR: Build failed: exit code 1

Relevant logs and/or screenshots

Logs from gitlab-ctl tail registry

2016-10-14_10:21:44.10675 time="2016-10-14T10:21:44.106701186Z" level=warning msg="error authorizing context: authorization token required" environment=production go.version=go1.5.4 http.request.host=registry.jakubigla.net http.request.id=76d63e53-f2cc-4d4e-995f-847b26a9b688 http.request.method=GET http.request.remoteaddr=52.29.148.147 http.request.uri="/v2/" http.request.useragent="docker/1.12.1 go/go1.6.3 git-commit/23cf638 kernel/3.13.0-92-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.1 \\(linux\\))" instance.id=5acdd224-a2c4-4005-b575-262dbf3a8b3f service=registry version=v2.4.1 
2016-10-14_10:21:44.10691 127.0.0.1 - - [14/Oct/2016:10:21:44 +0000] "GET /v2/ HTTP/1.0" 401 87 "" "docker/1.12.1 go/go1.6.3 git-commit/23cf638 kernel/3.13.0-92-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.1 \\(linux\\))"
2016-10-14_10:21:44.16280 time="2016-10-14T10:21:44.162756589Z" level=info msg="response completed" environment=production go.version=go1.5.4 http.request.host=registry.jakubigla.net http.request.id=66ec99f8-2148-400e-b388-68f091db47cb http.request.method=GET http.request.remoteaddr=52.29.148.147 http.request.uri="/v2/" http.request.useragent="docker/1.12.1 go/go1.6.3 git-commit/23cf638 kernel/3.13.0-92-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.1 \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.877973ms http.response.status=200 http.response.written=2 instance.id=5acdd224-a2c4-4005-b575-262dbf3a8b3f service=registry version=v2.4.1 
2016-10-14_10:21:44.16294 127.0.0.1 - - [14/Oct/2016:10:21:44 +0000] "GET /v2/ HTTP/1.0" 200 2 "" "docker/1.12.1 go/go1.6.3 git-commit/23cf638 kernel/3.13.0-92-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.1 \\(linux\\))"
2016-10-14_10:21:44.22594 time="2016-10-14T10:21:44.225894497Z" level=warning msg="error authorizing context: authorization token required" environment=production go.version=go1.5.4 http.request.host=registry.jakubigla.net http.request.id=4d38723c-fbfc-4ed3-9654-3f20114256f6 http.request.method=GET http.request.remoteaddr=52.29.148.147 http.request.uri="/v2/" http.request.useragent="docker/1.12.1 go/go1.6.3 git-commit/23cf638 kernel/3.13.0-92-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.1 \\(linux\\))" instance.id=5acdd224-a2c4-4005-b575-262dbf3a8b3f service=registry version=v2.4.1 
2016-10-14_10:21:44.22608 127.0.0.1 - - [14/Oct/2016:10:21:44 +0000] "GET /v2/ HTTP/1.0" 401 87 "" "docker/1.12.1 go/go1.6.3 git-commit/23cf638 kernel/3.13.0-92-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.1 \\(linux\\))"
2016-10-14_10:21:44.28875 time="2016-10-14T10:21:44.288708297Z" level=warning msg="error authorizing context: insufficient scope" environment=production go.version=go1.5.4 http.request.host=registry.jakubigla.net http.request.id=4f67213d-ae9c-4c3c-a1ce-db932ff86994 http.request.method=POST http.request.remoteaddr=52.29.148.147 http.request.uri="/v2/test/test/blobs/uploads/" http.request.useragent="docker/1.12.1 go/go1.6.3 git-commit/23cf638 kernel/3.13.0-92-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.1 \\(linux\\))" instance.id=5acdd224-a2c4-4005-b575-262dbf3a8b3f service=registry vars.name="test/test" version=v2.4.1 
2016-10-14_10:21:44.28891 127.0.0.1 - - [14/Oct/2016:10:21:44 +0000] "POST /v2/test/test/blobs/uploads/ HTTP/1.0" 401 198 "" "docker/1.12.1 go/go1.6.3 git-commit/23cf638 kernel/3.13.0-92-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.1 \\(linux\\))"

Gitlab runner info:

Version:      1.6.0
Git revision: 01b3ea1
Git branch:   1-6-stable
GO version:   go1.6.3
Built:        Thu, 22 Sep 2016 17:47:32 +0000
OS/Arch:      linux/amd64

Output of checks

Results of GitLab application Check

Checking GitLab Shell ...

GitLab Shell version >= 3.6.1 ? ... OK (3.6.1)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ... 
2/1 ... ok
2/9 ... ok
2/10 ... ok
2/11 ... ok
2/12 ... ok
5/13 ... ok
5/14 ... ok
5/15 ... ok
5/16 ... ok
5/17 ... ok
6/18 ... ok
7/19 ... repository is empty
7/20 ... repository is empty
7/21 ... ok
7/22 ... ok
8/23 ... ok
12/24 ... repository is empty
12/25 ... ok
10/26 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Reply by email ...

Reply by email is disabled in config/gitlab.yml

Checking Reply by email ... Finished

Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... no
  Try fixing it:
  sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads
  sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} \;
  sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} \;
  For more information see:
  doc/install/installation.md in section "GitLab"
  Please fix the error above and rerun the checks.
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ... 
2/1 ... yes
2/9 ... yes
2/10 ... yes
2/11 ... yes
2/12 ... yes
5/13 ... yes
5/14 ... yes
5/15 ... yes
5/16 ... yes
5/17 ... yes
6/18 ... yes
7/19 ... yes
7/20 ... yes
7/21 ... yes
7/22 ... yes
8/23 ... yes
12/24 ... yes
12/25 ... yes
10/26 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.1)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.7.4)
Active users: 7

Checking GitLab ... Finished

Results of GitLab environment info

System information
System:         Ubuntu 14.04
Current User:   git
Using RVM:      no
Ruby Version:   2.3.1p112
Gem Version:    2.6.6
Bundler Version:1.13.1
Rake Version:   10.5.0
Sidekiq Version:4.1.4

GitLab information
Version:        8.12.6
Revision:       fa20b51
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            https://gitlab.jakubigla.net
HTTP Clone URL: https://gitlab.jakubigla.net/some-group/some-project.git
SSH Clone URL:  git@gitlab.jakubigla.net:some-group/some-project.git
Using LDAP:     no
Using Omniauth: no

GitLab Shell
Version:        3.6.1
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
Hooks:          /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git:            /opt/gitlab/embedded/bin/git

Possible fixes

It must be losing auth header or something. I don't think it's a permissions problem