Can't push to gitlab registry, even though authentication via docker login was successful
Summary
After enabling private Gitlab registry (Omnibus installation), sorting out self signed certs problem (my registry is under different domain than gitlab itself), I'm able to login via docker login
, but no luck with docker push
command. This happens in Gitlab CI (auth via gitlab-ci-token) and in my dev environment (auth via gitlab credentials).
Steps to reproduce
My gitlab.rb
git_data_dir "/var/opt/gitlab/git-data"
external_url 'https://gitlab.jakubigla.net'
gitlab_rails['time_zone'] = 'UTC'
gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['gitlab_email_from'] = "gitlab@gitlab.jakubigla.net"
gitlab_rails['gitlab_email_display_name'] = "Gitlab"
gitlab_rails['gitlab_email_reply_to'] = "gitlab@gitlab.jakubigla.net"
gitlab_rails['gitlab_default_projects_features_container_registry'] = true
registry_external_url "https://registry.jakubigla.net"
nginx['redirect_http_to_https'] = true
nginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab.jakubigla.net.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab.jakubigla.net.key"
registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/registry.jakubigla.net.crt"
registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/registry.jakubigla.net.key"
Sample .gitlab-ci.yml
variables:
DOCKER_REGISTRY: "registry.jakubigla.net"
NAMESPACE: "test"
NAME: "test"
build:
image: docker:latest
stage: build
before_script:
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $DOCKER_REGISTRY
script:
- echo "FROM alpine" > Dockerfile
- docker build -t "$DOCKER_REGISTRY/$NAMESPACE/$NAME" .
- docker push "$DOCKER_REGISTRY/$NAMESPACE/$NAME"
tags:
- docker
only:
- master
Expected behavior
I expect that my freshly built image, can be pushed to my registry with no issues whatsoever, after a successful authentication.
Actual behavior
docker push
command throws an error:
unauthorized: authentication required
ERROR: Build failed: exit code 1
Relevant logs and/or screenshots
Logs from gitlab-ctl tail registry
2016-10-14_10:21:44.10675 time="2016-10-14T10:21:44.106701186Z" level=warning msg="error authorizing context: authorization token required" environment=production go.version=go1.5.4 http.request.host=registry.jakubigla.net http.request.id=76d63e53-f2cc-4d4e-995f-847b26a9b688 http.request.method=GET http.request.remoteaddr=52.29.148.147 http.request.uri="/v2/" http.request.useragent="docker/1.12.1 go/go1.6.3 git-commit/23cf638 kernel/3.13.0-92-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.1 \\(linux\\))" instance.id=5acdd224-a2c4-4005-b575-262dbf3a8b3f service=registry version=v2.4.1
2016-10-14_10:21:44.10691 127.0.0.1 - - [14/Oct/2016:10:21:44 +0000] "GET /v2/ HTTP/1.0" 401 87 "" "docker/1.12.1 go/go1.6.3 git-commit/23cf638 kernel/3.13.0-92-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.1 \\(linux\\))"
2016-10-14_10:21:44.16280 time="2016-10-14T10:21:44.162756589Z" level=info msg="response completed" environment=production go.version=go1.5.4 http.request.host=registry.jakubigla.net http.request.id=66ec99f8-2148-400e-b388-68f091db47cb http.request.method=GET http.request.remoteaddr=52.29.148.147 http.request.uri="/v2/" http.request.useragent="docker/1.12.1 go/go1.6.3 git-commit/23cf638 kernel/3.13.0-92-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.1 \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.877973ms http.response.status=200 http.response.written=2 instance.id=5acdd224-a2c4-4005-b575-262dbf3a8b3f service=registry version=v2.4.1
2016-10-14_10:21:44.16294 127.0.0.1 - - [14/Oct/2016:10:21:44 +0000] "GET /v2/ HTTP/1.0" 200 2 "" "docker/1.12.1 go/go1.6.3 git-commit/23cf638 kernel/3.13.0-92-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.1 \\(linux\\))"
2016-10-14_10:21:44.22594 time="2016-10-14T10:21:44.225894497Z" level=warning msg="error authorizing context: authorization token required" environment=production go.version=go1.5.4 http.request.host=registry.jakubigla.net http.request.id=4d38723c-fbfc-4ed3-9654-3f20114256f6 http.request.method=GET http.request.remoteaddr=52.29.148.147 http.request.uri="/v2/" http.request.useragent="docker/1.12.1 go/go1.6.3 git-commit/23cf638 kernel/3.13.0-92-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.1 \\(linux\\))" instance.id=5acdd224-a2c4-4005-b575-262dbf3a8b3f service=registry version=v2.4.1
2016-10-14_10:21:44.22608 127.0.0.1 - - [14/Oct/2016:10:21:44 +0000] "GET /v2/ HTTP/1.0" 401 87 "" "docker/1.12.1 go/go1.6.3 git-commit/23cf638 kernel/3.13.0-92-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.1 \\(linux\\))"
2016-10-14_10:21:44.28875 time="2016-10-14T10:21:44.288708297Z" level=warning msg="error authorizing context: insufficient scope" environment=production go.version=go1.5.4 http.request.host=registry.jakubigla.net http.request.id=4f67213d-ae9c-4c3c-a1ce-db932ff86994 http.request.method=POST http.request.remoteaddr=52.29.148.147 http.request.uri="/v2/test/test/blobs/uploads/" http.request.useragent="docker/1.12.1 go/go1.6.3 git-commit/23cf638 kernel/3.13.0-92-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.1 \\(linux\\))" instance.id=5acdd224-a2c4-4005-b575-262dbf3a8b3f service=registry vars.name="test/test" version=v2.4.1
2016-10-14_10:21:44.28891 127.0.0.1 - - [14/Oct/2016:10:21:44 +0000] "POST /v2/test/test/blobs/uploads/ HTTP/1.0" 401 198 "" "docker/1.12.1 go/go1.6.3 git-commit/23cf638 kernel/3.13.0-92-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.1 \\(linux\\))"
Gitlab runner info:
Version: 1.6.0
Git revision: 01b3ea1
Git branch: 1-6-stable
GO version: go1.6.3
Built: Thu, 22 Sep 2016 17:47:32 +0000
OS/Arch: linux/amd64
Output of checks
Results of GitLab application Check
Checking GitLab Shell ...
GitLab Shell version >= 3.6.1 ? ... OK (3.6.1)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
2/1 ... ok
2/9 ... ok
2/10 ... ok
2/11 ... ok
2/12 ... ok
5/13 ... ok
5/14 ... ok
5/15 ... ok
5/16 ... ok
5/17 ... ok
6/18 ... ok
7/19 ... repository is empty
7/20 ... repository is empty
7/21 ... ok
7/22 ... ok
8/23 ... ok
12/24 ... repository is empty
12/25 ... ok
10/26 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... no
Try fixing it:
sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads
sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} \;
sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} \;
For more information see:
doc/install/installation.md in section "GitLab"
Please fix the error above and rerun the checks.
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
2/1 ... yes
2/9 ... yes
2/10 ... yes
2/11 ... yes
2/12 ... yes
5/13 ... yes
5/14 ... yes
5/15 ... yes
5/16 ... yes
5/17 ... yes
6/18 ... yes
7/19 ... yes
7/20 ... yes
7/21 ... yes
7/22 ... yes
8/23 ... yes
12/24 ... yes
12/25 ... yes
10/26 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.1)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.7.4)
Active users: 7
Checking GitLab ... Finished
Results of GitLab environment info
System information
System: Ubuntu 14.04
Current User: git
Using RVM: no
Ruby Version: 2.3.1p112
Gem Version: 2.6.6
Bundler Version:1.13.1
Rake Version: 10.5.0
Sidekiq Version:4.1.4
GitLab information
Version: 8.12.6
Revision: fa20b51
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL: https://gitlab.jakubigla.net
HTTP Clone URL: https://gitlab.jakubigla.net/some-group/some-project.git
SSH Clone URL: git@gitlab.jakubigla.net:some-group/some-project.git
Using LDAP: no
Using Omniauth: no
GitLab Shell
Version: 3.6.1
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git: /opt/gitlab/embedded/bin/git
Possible fixes
It must be losing auth header or something. I don't think it's a permissions problem