Failed to connect to gitlab.domain.com port 443: Connection refused
Summary
I am running GitLab Enterprise Edition 11.9.0-ee on a server in my local network on a CentOS 7 VM. We are currently evaluating it. This was installed using omnibus installer as suggested in docs. It was setup with a self-signed cert initially and later (as mentioned below), the cert was updated to one signed by an internal CA in our org. I have also installed gitlab-runner on the same machine using the recommended steps
I registered the runner with docker as my executor as mentioned on this page
I was successfully able to run a pipeline to provision a terraform:light container and run some code in it, up until last week. After that I updated the certificate with a new one (internal CA) on the gitlab instance and was able to bring it up with no issues. Everything checked out in the browser as well.
But then I had to unregister and register a new runner as it couldn't it didn't recognize the certs. I removed and added the new cert in /etc/gitlab-runner/certs/ and restarted the service. And started facing the current issue where the docker container fails to clone the repo. I searched a lot through the forums and also tried a lot of suggested fixes, but I haven't been able to fix it.
Steps to reproduce
- Change certificate for main gitlab instance to one signed by internal CA. Update nginx using the command mentioned here after the update. Also reconfigure using gitlab-ctl
- Unregister and register a new runner (on the same instance where gitlab is setup) since the previous one wouldn't be able to talk to to gitlab using the steps mentioned on this page.
- Commit some code to trigger the pipeline.
- Observer failure in the logs when git tries to clone the repo.
Actual behavior
Git clone fails with the error - Failed to connect to gitlab.domain.com port 443: Connection refused
Expected behavior
Git clone shouldn't fail
Relevant logs and/or screenshots
.gitlab-ci.yml:
image:
name: hashicorp/terraform:light
entrypoint:
- '/usr/bin/env'
- 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
before_script:
- rm -rf .terraform
- terraform --version
- terraform init
stages:
- validate
- plan
- apply
validate:
somecode
plan:
somecode
apply:
somecode
Logs:
Running with gitlab-runner 11.9.0 (692ae235)
on runnerdind hymEtVDV
Using Docker executor with image hashicorp/terraform:light ...
Pulling docker image hashicorp/terraform:light ...
Using docker image sha256:7cd543b4280182930cfeeb9af69dddc5ac6af3de86987d9d9bca86b3cc4c48dc for hashicorp/terraform:light ...
Running on runner-hymEtVDV-project-5-concurrent-0 via gitlab...
Initialized empty Git repository in /builds/group/project-name/.git/
Fetching changes...
Created fresh repository.
fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.domain.com/group/project-name/.git/': Failed to connect to gitlab.domain.com port 443: Connection refused
ERROR: Job failed: exit code 1
Environment description
On prem installation, Docker executor:
# docker info
Containers: 10
Running: 0
Paused: 0
Stopped: 10
Images: 4
Server Version: 18.09.3
Storage Driver: devicemapper
Pool Name: docker-253:1-138911062-pool
Pool Blocksize: 65.54kB
Base Device Size: 10.74GB
Backing Filesystem: xfs
Udev Sync Supported: true
Data file: /dev/loop0
Metadata file: /dev/loop1
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Data Space Used: 1.108GB
Data Space Total: 107.4GB
Data Space Available: 47.22GB
Metadata Space Used: 18.45MB
Metadata Space Total: 2.147GB
Metadata Space Available: 2.129GB
Thin Pool Minimum Free Space: 10.74GB
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.149-RHEL7 (2018-07-20)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: e6b3f5632f50dbc4e9cb6288d911bf4f5e95b18e
runc version: 6635b4f0c6af3810594d2770f662f34ddc15b40d
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-957.10.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.701GiB
Name: gitlab
ID: JF45:NXCE:GMGN:OTNP:YLIJ:ZMXC:ANN6:6PIO:5BKW:SHUI:DCQD:CI5D
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: the devicemapper storage-driver is deprecated, and will be removed in a future release.
WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Used GitLab Runner version
Version: 11.9.0 Git revision: 692ae235 Git branch: 11-9-stable GO version: go1.8.7 Built: 2019-03-22T12:02:58+0000 OS/Arch: linux/amd64