Docker login fails to gitlab registry, when gitlab and the registry are on separate domains

Summary

When using different domains for GitLab and the bundled docker registry, I cannot docker login to the registry, due to a request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) (Client.Timeout exceeded while awaiting headers) error.

Steps to reproduce

Set external url to some private FQDN

external_url http://gitlab.mydomain.com # This DNS name resolves to a private IP address that only works within our local VPN.

Set registry_external_url to some public FQDN

registry_external_url 'http://docker.mydomain.com:5005' # This DNS name is a public one and resolves to a publicly accessible host.  
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "docker.mydomain.com"
gitlab_rails['registry_port'] = "5005"

Try to login:

docker login -u myuser -p "mypass" http://docker.mydomain.com:5005

This fails with:

What is the current bug behavior?

Docker login fails with:

Error response from daemon: Get http://docker.mydomain.com:5005/v2/: Get http://gitlab.mydomain.com/jwt/auth?account=myuser&client_id=docker&offline_token=true&service=container_registry: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) (Client.Timeout exceeded while awaiting headers)

What is the expected correct behavior?

Docker login should succeed, with the below message:

Login Succeeded

Relevant logs and/or screenshots

(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's very hard to read otherwise.)

Output of checks

docker -v
Docker version 1.13.1, build 092cba3

Results of GitLab environment info

Expand for output related to GitLab environment info

docker exec gitlab gitlab-rake gitlab:env:info

System information
System:		
Current User:	git
Using RVM:	no
Ruby Version:	2.3.3p222
Gem Version:	2.6.6
Bundler Version:1.13.7
Rake Version:	12.0.0
Redis Version:	3.2.5
Git Version:	2.13.5
Sidekiq Version:5.0.4
Go Version:	unknown

GitLab information
Version:	9.5.4
Revision:	fbffc27
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
URL:		http://gitlab.mydomain.com
HTTP Clone URL:	http://gitlab.mydomain.com/some-group/some-project.git
SSH Clone URL:	git@gitlab.mydomain.com:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	no

GitLab Shell
Version:	5.8.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
Hooks:		/opt/gitlab/embedded/service/gitlab-shell/hooks
Git:		/opt/gitlab/embedded/bin/git

</pre>
</details>

Results of GitLab application Check

Expand for output related to the GitLab application check


docker exec gitlab gitlab-rake gitlab:check SANITIZE=true
Checking GitLab Shell ...

GitLab Shell version >= 5.8.0 ? ... OK (5.8.0)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ... 
4/2 ... repository is empty
6/3 ... ok
6/4 ... ok
6/5 ... ok
2/7 ... ok
2/8 ... ok
2/9 ... ok
2/10 ... ok
2/11 ... ok
2/12 ... ok
2/13 ... ok
2/14 ... ok
2/15 ... ok
2/16 ... ok
2/17 ... ok
2/18 ... ok
2/19 ... ok
2/20 ... ok
2/21 ... ok
2/22 ... ok
2/23 ... ok
2/24 ... ok
2/25 ... ok
2/26 ... ok
11/27 ... ok
14/28 ... ok
14/29 ... ok
2/30 ... ok
15/31 ... ok
11/32 ... ok
2/33 ... ok
2/34 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Reply by email ...

Reply by email is disabled in config/gitlab.yml

Checking Reply by email ... Finished

Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ... 
4/2 ... yes
6/3 ... yes
6/4 ... yes
6/5 ... yes
2/7 ... yes
2/8 ... yes
2/9 ... yes
2/10 ... yes
2/11 ... yes
2/12 ... yes
2/13 ... yes
2/14 ... yes
2/15 ... yes
2/16 ... yes
2/17 ... yes
2/18 ... yes
2/19 ... yes
2/20 ... yes
2/21 ... yes
2/22 ... yes
2/23 ... yes
2/24 ... yes
2/25 ... yes
2/26 ... yes
11/27 ... yes
14/28 ... yes
14/29 ... yes
2/30 ... yes
15/31 ... yes
11/32 ... yes
2/33 ... yes
2/34 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.3 ? ... yes (2.3.3)
Git version >= 2.7.3 ? ... yes (2.13.5)
Active users: ... 11

Checking GitLab ... Finished

Possible fixes

Even though this behaviour of having GitLab and the Registry on different domains seems to be supported, it seems to me this does not cover the case where the registry is on a publicly accessible URL and GitLab isn't.

From what I can tell, the error message seems to already hint to the direction of the problem.

Even though the docker registry is on the docker.mydomain.com:5005 url, from the error message it seems GitLab is using its own external_url for the /jwt/auth URL, when it probably should be using the docker registry one...?