Not able to connect to Docker Hosted Registry - getsockopt: connection refused

Hi,

I am currently hosting a copy of Gitlab-ee in an Docker container in Rancher, and am having issues with the registry.

I followed this guide:

http://rancher.com/how-to-run-gitlab-in-rancher-1/

and have the following setup:

2x domains: gitlab.xxxxxxx.co.uk and registry.gitlab.xxxxxxx.co.uk both resolve to the same IP address

In gitlab environment on rancher:

Lets encrypt, which has generated SSL certs for 2 domains: gitlab.xxxxxx.co.uk && registry.gitlab.xxxxxx.co.uk

Load Balancer with the following rules:

public -> HTTP -> gitlab.xxxxxx.co.uk -> port 80 -> {gitlab stack} -> port 80
public -> HTTPS -> gitlab.xxxxxx.co.uk -> port 443 -> {gitlab stack} -> port 80
public -> HTTPS -> registry.gitlab.xxxxxx.co.uk -> port 443 -> {gitlab stack} -> port 80
public -> TCP -> port 22 -> {gitlab stack} -> port 22

Gitlab install.

This has been configured with the following values in the gitlab.rb file

external_url 'https://gitlab.xxxxxx.co.uk'
gitlab_rails['gitlab_ssh_host'] = "gitlab.xxxxxx.co.uk"
gitlab_rails['db_adapter'] = "postgresql"
gitlab_rails['db_encoding']="unicode"
gitlab_rails['db_database'] = "gitlab_production"
gitlab_rails['db_username'] ="gitlab"
gitlab_rails['db_password'] = "{REDACTED}"
gitlab_rails['db_host'] = "{REDACTED - but Amazon RDS}"
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "{REDACTED}"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "{REDACTED}"
gitlab_rails['smtp_password'] = "{REDACTED}"
gitlab_rails['smtp_domain'] = "xxxxxx.co.uk"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
registry_external_url 'https://registry.gitlab.xxxxxx.co.uk'
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "registry.gitlab.xxxxxxx.co.uk"
gitlab_rails['registry_api_url'] = 'http://localhost:5000'
gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"
gitlab_rails['registry_key_path'] = '/var/opt/gitlab/gitlab-rails/certificate.key'
gitlab_rails['registry_issuer'] = 'omnibus-gitlab-issuer'
registry['enable'] = true
registry['token_realm'] = 'https://gitlab.xxxxxxxx.co.uk'
nginx['listen_port'] = 80
nginx['listen_https'] = false
nginx['proxy_set_headers'] = {'Host' => '$http_host_with_default','X-Real-IP' => '$remote_addr','X-Forwarded-For' => '$proxy_add_x_forwarded_for','X-Forwarded-Proto' => 'https','X-Forwarded-Ssl' => 'on','Upgrade' => '$http_upgrade','Connection' => '$connection_upgrade' }
registry_nginx['enable'] = true
registry_nginx['listen_port'] = 80
registry_nginx['listen_https'] = false
registry_nginx['proxy_set_headers'] = {'Host' => '$http_host','X-Real-IP' => '$remote_addr','X-Forwarded-For' => '$proxy_add_x_forwarded_for','X-Forwarded-Proto' => 'https','X-Forwarded-Ssl' => 'on'}
registry_nginx['custom_gitlab_server_config'] = 'proxy_cache_convert_head off;'

The site works fine, and I can create repos etc via SSL.

Then in another environment I have a gitlab runner in a stack with the following config in config.toml:

concurrent = 1
check_interval = 0

[[runners]]
  name = "runner-rancher-cattle"
  url = "https://gitlab.xxxxxx.co.uk"
  token = "{REDACTED}"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "docker:stable"
    privileged = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
    shm_size = 0
  [runners.cache]

When pushing a project with the following .gitlab-ci.yml The runner can run test tasks just fine, and build docker images, but when it comes to pushing to the docker registry it fails:

variables:
  IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG

stages:
  - test
  - create

test:
  stage: test
  image: ruby:2.4.2-alpine
  script:
    - apk add --update  build-base libxml2-dev libxslt-dev postgresql-dev && rm -rf /var/cache/apk/*
    - bundle install --without development
    - bundle exec rspec --format documentation

create:
  stage: create
  image: docker:latest
  only:
    - tags
  before_script:
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.xxxxxxxx.co.uk
  script:
    - docker build -t $IMAGE_TAG .
    - docker push $IMAGE_TAG

When pushing to the registry:

Running with gitlab-runner 10.1.0 (c1ecf97f)
  on runner-rancher-cattle (22680871)
Using Docker executor with image docker:latest ...
Using docker image sha256:330def6290307c5144addaaff7c1c8dc6b7f2e49846590b0978e890cc6ca3705 for predefined container...
Pulling docker image docker:latest ...
Using docker image docker:latest ID=sha256:5352373707c92f546d3b519d61ce5a2335d67cf06113a4223813ac7fe789f13a for build container...
Running on runner-22680871-project-1-concurrent-0 via 91bcad60295a...
Fetching changes...
HEAD is now at f04ba03 changed registery location
Checking out f04ba03d as v1.0rc2...
Skipping Git submodules setup
$ docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.xxxxxxx.co.uk
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded
$ docker build -t $IMAGE_TAG .
Sending build context to Docker daemon  122.9kB

Step 1/8 : FROM ruby:2.4.2-alpine
 ---> 085b40166989
Step 2/8 : RUN apk add --update  build-base libxml2-dev libxslt-dev postgresql-dev && rm -rf /var/cache/apk/*
 ---> Using cache
 ---> a36dbcac0c7d
Step 3/8 : WORKDIR /app
 ---> Using cache
 ---> c7b86dfbf482
Step 4/8 : COPY Gemfile Gemfile.lock /app/
 ---> Using cache
 ---> 099b84650e6e
Step 5/8 : RUN bundle install --without test development
 ---> Using cache
 ---> 41bf4f025049
Step 6/8 : RUN bundle config disable_exec_load true
 ---> Using cache
 ---> 36b6a8bf5d67
Step 7/8 : COPY . /app
 ---> 54cdf4d2c496
Removing intermediate container 474c6cf4d3a6
Step 8/8 : CMD cd /app && bundle exec rackup
 ---> Running in 50d7714cd215
 ---> 023041c0ec06
Removing intermediate container 50d7714cd215
Successfully built 023041c0ec06
$ docker push $IMAGE_TAG
The push refers to a repository [registry.gitlab.xxxxx.co.uk:5000/xxxxxx/test-api-project]
Get https://registry.gitlab.xxxxxx.co.uk:5000/v1/_ping: dial tcp x.x.x.x:5000: getsockopt: connection refused
ERROR: Job failed: exit code 1

or spefically the line: Get https://registry.gitlab.xxxxxx.co.uk:5000/v1/_ping: dial tcp x.x.x.x:5000: getsockopt: connection refused

However, it seems to login in ok? $ docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.xxxxxxx.co.uk WARNING! Using --password via the CLI is insecure. Use --password-stdin. Login Succeeded

I ran gitlab-rake gitlab:check to check the install and this is its output:

Checking GitLab Shell ...

GitLab Shell version >= 5.9.3 ? ... OK (5.9.3)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
safetytoolbox / test-api-project ... ok
Api / stb_background_worker ... repository is empty
Api / stb_api ... repository is empty
safetytoolbox / gitlab_runner ... repository is empty
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK

Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Reply by email is disabled in config/gitlab.yml
Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
safetytoolbox / test-api-project ... yes
Api / stb_background_worker ... yes
Api / stb_api ... yes
safetytoolbox / gitlab_runner ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.3 ? ... yes (2.3.5)
Git version >= 2.7.3 ? ... yes (2.13.6)
Git user has default SSH configuration? ... yes
Active users: ... 3
Elasticsearch version 5.1 - 5.5? ... skipped (elasticsearch is disabled)

Checking GitLab ... Finished

Double checked ports on AWS, and have port 5000 open on the gitlab install.