Skip to content

Failed to connect to gitlab.domain.com port 443: Connection refused

Summary

I am running GitLab Enterprise Edition 11.9.0-ee on a server in my local network on a CentOS 7 VM. We are currently evaluating it. This was installed using omnibus installer as suggested in docs. It was setup with a self-signed cert initially and later (as mentioned below), the cert was updated to one signed by an internal CA in our org. I have also installed gitlab-runner on the same machine using the recommended steps

I registered the runner with docker as my executor as mentioned on this page

I was successfully able to run a pipeline to provision a terraform:light container and run some code in it, up until last week. After that I updated the certificate with a new one (internal CA) on the gitlab instance and was able to bring it up with no issues. Everything checked out in the browser as well.

But then I had to unregister and register a new runner as it couldn't it didn't recognize the certs. I removed and added the new cert in /etc/gitlab-runner/certs/ and restarted the service. And started facing the current issue where the docker container fails to clone the repo. I searched a lot through the forums and also tried a lot of suggested fixes, but I haven't been able to fix it.

Steps to reproduce

  1. Change certificate for main gitlab instance to one signed by internal CA. Update nginx using the command mentioned here after the update. Also reconfigure using gitlab-ctl
  2. Unregister and register a new runner (on the same instance where gitlab is setup) since the previous one wouldn't be able to talk to to gitlab using the steps mentioned on this page.
  3. Commit some code to trigger the pipeline.
  4. Observer failure in the logs when git tries to clone the repo.

Actual behavior

Git clone fails with the error - Failed to connect to gitlab.domain.com port 443: Connection refused

Expected behavior

Git clone shouldn't fail

Relevant logs and/or screenshots

.gitlab-ci.yml:

image:
  name: hashicorp/terraform:light
  entrypoint:
    - '/usr/bin/env'
    - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'

before_script:
  - rm -rf .terraform
  - terraform --version
  - terraform init

stages:
  - validate
  - plan
  - apply

validate:
  somecode

plan:
  somecode
apply:
  somecode

Logs:

Running with gitlab-runner 11.9.0 (692ae235)
  on runnerdind hymEtVDV
Using Docker executor with image hashicorp/terraform:light ...
Pulling docker image hashicorp/terraform:light ...
Using docker image sha256:7cd543b4280182930cfeeb9af69dddc5ac6af3de86987d9d9bca86b3cc4c48dc for hashicorp/terraform:light ...
Running on runner-hymEtVDV-project-5-concurrent-0 via gitlab...
Initialized empty Git repository in /builds/group/project-name/.git/
Fetching changes...
Created fresh repository.
fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.domain.com/group/project-name/.git/': Failed to connect to gitlab.domain.com port 443: Connection refused
ERROR: Job failed: exit code 1

Environment description

On prem installation, Docker executor:

# docker info
Containers: 10
 Running: 0
 Paused: 0
 Stopped: 10
Images: 4
Server Version: 18.09.3
Storage Driver: devicemapper
 Pool Name: docker-253:1-138911062-pool
 Pool Blocksize: 65.54kB
 Base Device Size: 10.74GB
 Backing Filesystem: xfs
 Udev Sync Supported: true
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 1.108GB
 Data Space Total: 107.4GB
 Data Space Available: 47.22GB
 Metadata Space Used: 18.45MB
 Metadata Space Total: 2.147GB
 Metadata Space Available: 2.129GB
 Thin Pool Minimum Free Space: 10.74GB
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.149-RHEL7 (2018-07-20)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: e6b3f5632f50dbc4e9cb6288d911bf4f5e95b18e
runc version: 6635b4f0c6af3810594d2770f662f34ddc15b40d
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-957.10.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.701GiB
Name: gitlab
ID: JF45:NXCE:GMGN:OTNP:YLIJ:ZMXC:ANN6:6PIO:5BKW:SHUI:DCQD:CI5D
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: the devicemapper storage-driver is deprecated, and will be removed in a future release.
WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
         Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.

Used GitLab Runner version

Version: 11.9.0 Git revision: 692ae235 Git branch: 11-9-stable GO version: go1.8.7 Built: 2019-03-22T12:02:58+0000 OS/Arch: linux/amd64