Shell GitLab runner behind proxy cannot connect to GitLab server

Summary

My shell GitLab Runner behind a corporate proxy doesn't connect with our GitLab server. Logs say

couldn't execute POST against https://[GITLAB SERVER NAME REDACTED]/api/v4/jobs/request: Post https://[REDACTED]/api/v4/jobs/request: dial tcp [GITLAB SERVER IP ADDRESS REDACTED]:443: i/o timeout

What surprises me is that the runner appears in GitLab. I.e. the runner communicated with GitLab after registration, but fails when checking for build jobs later.

Very important: The IP address in the log ("dial tcp [IP ADDRESS REDACTED]:443") belongs to the GitLab server. If GitLab Runner used the proxy correctly I'd expect the IP address of the proxy here, as we are on TCP level here.

Steps to reproduce

  • add proxy settings in etc/environment 
    HTTP_PROXY=http://[REDACTED]:8080
HTTPS_PROXY=https://[REDACTED]:8080
  • add proxy environment variables in /etc/profile.d/myenvvars.sh 
    export HTTP_PROXY=http://[REDACTED]:8080
export HTTPS_PROXY=https://[REDACTED]:8080
  • add proxy environment variables to /etc/gitlab-runner/config.toml 
    [[runners]]
    environment = ["HTTPS_PROXY=https://[REDACTED]:8080", "HTTP_PROXY=http://[REDACTED]:8080"]
  • register a shell gitlab-runner on a RHEL-6 system behind a corporate proxy
  • check if proxy env variables are available for the gitlab-runner daemon 
    $ runuser -s /bin/bash gitlab-runner -c "echo $HTTPS_PROXY"
https://[REDACTED]:8080
  • check if proxy env works for other programs 
    $ curl https://www.google.com
  • gitlab-runner start
  • reboot (just to ensure, gitlab runner wasn't started in an environment where the proxy settings hadn't been available)
  • Curl to the failed URL to ensure that there is no problem with the server 
    $ curl https://[REDACTED]/api/v4/jobs/request
{"error":"404 Not Found"}
  • run tail -f /var/log/messages|grep gitlab-runner to watch the log

Actual behavior

  • the runner appears with an exclamation mark in GitLab
  • GitLab says "New runner. Has not connected yet."
  • The runner logs reveal that the runner's HTTP requests result in a timeout
  • The runner doesn't process build jobs

Expected behavior

  • The runner appears with a green circle in the gitlab web ui
  • The runner processes build jobs with matching tags

Relevant logs and/or screenshots

Oct 18 08:21:55 xl126 gitlab-runner[2334]: time="2018-10-18T08:21:55+02:00" level=warning msg="Checking for jobs... failed" runner=b94d7e0f status="couldn't execute POST against https://[REDACTED]/api/v4/jobs/request: Post https://[REDACTED]/api/v4/jobs/request: dial tcp [REDACTED]:443: i/o timeout"

image

Environment description

  • RedHat Enterprise Linux 6 (RHEL-6)
  • Docker is not used (it's not supported by RHEL-6)
  • Self hosted GitLab CE
  • RHEL-6 doesn't use systemd, so I cannot use the systemd drop in as described in the docs

Used GitLab Runner version

Version:      11.3.1~beta.4.g0aa5179e
Git revision: 0aa5179e
Git branch:   11-3-stable
GO version:   go1.8.7
Built:        2018-09-24T16:02:32+0000
OS/Arch:      linux/amd64

Please run and paste the output of gitlab-runner --version. If you are using a Runner where you don't have access to, please paste at least the first lines the from build log, like:

Running with gitlab-ci-multi-runner 1.4.2 (bcc1794)
Using Docker executor with image golang:1.8 ...
Edited by Ulrich Hecht