Gitlab Registry fails to authenticate with docker-machine

Summary

I'm trying to build a Docker image on a docker-machine host and push that image to Gitlab registry. I enabled the registry, then used docker-machine as described HERE (with the exception that I used 172.17.0.1 as host IP, but it worked after a trivial change in Gitlab Runner configuration). The problem is that after building the image, when I try to push it, I get the following error:

    $ docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN gitlab.organization.com:39140
    Login Succeeded
    $ docker build -t gitlab.organization.com:39140/organization/project-name .
    (build went fine)
    $ docker push gitlab.organization.com:39140/organization/project-name
    The push refers to a repository [gitlab.organization.com:39140/organization/project-name]
    9da290505e25: Preparing
    4edf62705ffc: Preparing
    9845b0e669c4: Preparing
    0dd280e9ab09: Preparing
    1efea06cfe5b: Preparing
    cf516324493c: Preparing
    cf516324493c: Waiting
    denied: access forbidden
    ERROR: Job failed: exit code 1

Steps to reproduce

  1. Set up docker-machine,
  2. Use the following .gitlab-ci.yml file (change host):
    image: my-base-image
    
    variables:
        GIT_SUBMODULE_STRATEGY: recursive
        DOCKER_TLS_VERIFY: "1"
        DOCKER_HOST: "tcp://172.17.0.1:2376"
        DOCKER_CERT_PATH: "certs"
    
    test:build:
      script:
      - mkdir $DOCKER_CERT_PATH
      - echo "$CA" > $DOCKER_CERT_PATH/ca.pem
      - echo "$CLIENT_CERT" > $DOCKER_CERT_PATH/cert.pem
      - echo "$CLIENT_KEY" > $DOCKER_CERT_PATH/key.pem\
      - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN gitlab.organization.com:39140
      - docker build -t gitlab.organization.com:39140/organization/project-name .
      - docker pull gitlab.organization.com:39140/organization/project-name

What is the current bug behavior?

Push/pull operations fail with "denied: access forbidden"

What is the expected correct behavior?

Push/pull succeed.

Relevant logs and/or screenshots

Here's a line from /var/log/gitlab/registry/current:

2017-09-29_11:32:21.23393 time="2017-09-29T06:32:21.233855102-05:00" level=warning msg="error authorizing context: authorization token required" environment=production go.version=go1.8.1 http.request.host="gitlab.organization.com:39140" http.request.id=de1f3cee-39ee-47c1-b75f-a2fec2f0a35d http.request.method=GET http.request.remoteaddr=52.41.80.63 http.request.uri="/v2/" http.request.useragent="docker/17.09.0-ce go/go1.8.3 git-commit/afdb6d4 kernel/4.4.0-1020-aws os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.6 \\(linux\\))" instance.id=20190cc9-98d7-4fe0-85ab-14845727dead service=registry version=v2.6.1-1-gdd544a8

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:         Debian 8.9
Current User:   git
Using RVM:      no
Ruby Version:   2.3.5p376
Gem Version:    2.6.13
Bundler Version:1.13.7
Rake Version:   12.0.0
Redis Version:  3.2.5
Git Version:    2.13.5
Sidekiq Version:5.0.4
Go Version:     unknown

GitLab information
Version:        10.0.2
Revision:       06a5a33
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            https://gitlab.profound.net
HTTP Clone URL: https://gitlab.profound.net/some-group/some-project.git
SSH Clone URL:  ssh://git@gitlab.profound.net:1027/some-group/some-project.git
Using LDAP:     no
Using Omniauth: no


GitLab Shell
Version:        5.9.0
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
root@gitlab:/var/opt/gitlab/registry# sudo gitlab-rake gitlab:check SANITIZE=true                                                      Checking GitLab Shell ...

GitLab Shell version >= 5.9.0 ? ... OK (5.9.0)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
3/2 ... ok
3/3 ... ok
3/4 ... ok
3/5 ... ok
3/7 ... ok
3/8 ... ok
3/9 ... ok
3/10 ... ok
3/11 ... ok
1/14 ... ok
3/15 ... ok
3/16 ... ok
3/17 ... ok
3/18 ... ok
3/19 ... ok
9/20 ... ok
3/21 ... ok
3/22 ... ok
3/23 ... repository is empty
3/24 ... ok
1/25 ... ok
3/26 ... ok
15/27 ... ok
15/28 ... ok
15/29 ... ok
15/30 ... ok
3/31 ... ok
15/32 ... ok
3/33 ... ok
3/34 ... ok
3/35 ... ok
1/36 ... ok
7/37 ... ok
3/38 ... ok
3/39 ... ok
3/40 ... ok
7/41 ... ok
15/43 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Sidekiq ...


Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Reply by email is disabled in config/gitlab.yml
Checking LDAP ...


LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
3/2 ... yes
3/3 ... yes
3/4 ... yes
3/5 ... yes
3/7 ... yes
3/8 ... yes
3/9 ... yes
3/10 ... yes
3/11 ... yes
1/14 ... yes
3/15 ... yes
3/16 ... yes
3/17 ... yes
3/18 ... yes
3/19 ... yes
9/20 ... yes
3/21 ... yes
3/22 ... yes
3/23 ... yes
3/24 ... yes
1/25 ... yes
3/26 ... yes
15/27 ... yes
15/28 ... yes
15/29 ... yes
15/30 ... yes
3/31 ... yes
15/32 ... yes
3/33 ... yes
3/34 ... yes
3/35 ... yes
1/36 ... yes
7/37 ... yes
3/38 ... yes
3/39 ... yes
3/40 ... yes
7/41 ... yes
15/43 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.3 ? ... yes (2.3.5)
Git version >= 2.7.3 ? ... yes (2.13.5)
Git user has default SSH configuration? ... yes
Active users: ... 9


Checking GitLab ... Finished

Possible fixes

Not sure. My theory is that certificate from $DOCKER_CERT_PATH is used both for connecting with docker-machine host and Gitlab registry. Is that possible? Can I force login/password authentication for Gitlab Registry or generate a key that CI could use?