GitLab runner proxy issues in Redhat Openshift pods

This is the issue faced by one of our Premium customers. They have GitLab SaaS(SAML SSO enabled) inside their VPN and trying to register GitLab runner on OpenShift Cluster for their deployments. Below are more more details on the issue.

Error message

fatal: unable to access 'https://gitlab.com/{project path}.git/': OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to gitlab.com:443

Full error log in GitLab console

1 Running with gitlab-runner 14.0.1 (c1edb478)

2 on devops-runner-runner-5bf47f8fc8-4bj95 cV451jf5

3 Resolving secrets00:00

5 Preparing the "kubernetes" executor00:00

6 Using Kubernetes namespace: gitlab-runners

7 Using Kubernetes executor with image alpine ...

8 Using attach strategy to execute scripts...

10 Preparing environment00:09

11 Waiting for pod gitlab-runners/runner-cv451jf5-project-28845083-concurrent-0c24d2 to be running, status is Pending

12 Waiting for pod gitlab-runners/runner-cv451jf5-project-28845083-concurrent-0c24d2 to be running, status is Pending

13  ContainersNotInitialized: "containers with incomplete status: [init-logs]"

14  ContainersNotReady: "containers with unready status: [build helper]"

15  ContainersNotReady: "containers with unready status: [build helper]"

16 Waiting for pod gitlab-runners/runner-cv451jf5-project-28845083-concurrent-0c24d2 to be running, status is Pending

17  ContainersNotReady: "containers with unready status: [build helper]"

18  ContainersNotReady: "containers with unready status: [build helper]"

19 Running on runner-cv451jf5-project-28845083-concurrent-0c24d2 via devops-runner-runner-5bf47f8fc8-4bj95...

21 Getting source from Git repository00:01

22 Fetching changes with git depth set to 50...

23 Initialized empty Git repository in /builds/cV451jf5/0/{project path}.git/

24 Created fresh repository.

25 fatal: unable to access 'https://gitlab.com/{project path}.git/': OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to gitlab.com:443

27 Cleaning up file based variables00:00

29 ERROR: Job failed: command terminated with exit code 1

OCP4

We are using the GitLab Runner 1.2.0 operator. All instructions were followed for basic set up. We also attempted to add a custom-env ConfigMap and mount that with our proxy but we did not have success. Here are the contents of the custom-env file:

kind: ConfigMap

apiVersion: v1

metadata:

  name: custom-env

  namespace: gitlab-runners

  …

data:

  HTTPS_PROXY: ‘[PLACEHOLDER]’

  HTTP_PROXY: ‘[PLACEHOLDER]’

  NO_PROXY: 'example.com,synergy.local,10.11.0.1'

  PRE_CLONE_SCRIPT: >-

    export http_proxy=[PLACEHOLDER];export https_proxy=[PLACEHOLDER]

We also have attempted to add our cert bundle to the configuration without success. We were able to capture the spawned runner pod and noticed that the proxy settings were not getting set, and that’s where we tried to set the PRE_CLONE_SCRIPT Also it was pointed out that the image may need the proxy set in the base image, but that was also unsuccessful. After reviewing the runners main ConfigMap the register-runner script looks to be the culprit of the issue and looks to be a bug. Here is the content of the register-runner piece of the yaml:

register-runner: |

    #!/bin/bash

    set -e

    MAX_REGISTER_ATTEMPTS=30

 

    setup_proxy() {

      input=$*

 

      if [[ -v HTTP_PROXY ]] && [[ ! -z HTTP_PROXY ]]; then

        input+=" --env \"HTTP_PROXY=${HTTP_PROXY}\""

        git_http_proxy="git config --global http.proxy $HTTP_PROXY"

      fi

 

      if [[ -v HTTPS_PROXY && ! -z HTTPS_PROXY ]]; then

        input+=" --env \"HTTPS_PROXY=${HTTPS_PROXY}\""

        git_https_proxy="git config --global https.proxy $HTTPS_PROXY"

      fi

 

      if [[ -v git_http_proxy ]] && [[ ! -z git_http_proxy ]];

      then

        pre_clone_script=$git_http_proxy

      fi

 

      if [[ ( -v pre_clone_script && ! -z pre_clone_script )  && \

      ( -v git_https_proxy && ! -z git_https_proxy ) ]];

      then

        pre_clone_script+="; $git_https_proxy"

      fi

 

      if [[ ! -v pre_clone_script && \

      ( -v git_https_proxy && ! -z git_https_proxy ) ]];

      then

        pre_clone_script="$git_https_proxy"

      fi

 

      if [[ -v pre_clone_script ]] && [[ ! -z pre_clone_script ]]

      then

        input+=" --pre-clone-script \"$pre_clone_script\""

      fi

 

      echo $input

    }

 

    register_command() {

      command="/entrypoint register --non-interactive"

 

      if [[ ( -v RUNNER_ENV  && ! -z RUNNER_ENV )  || \

      ( -v RUNNER_PRE_CLONE_SCRIPT  && ! -z RUNNER_PRE_CLONE_SCRIPT  ) ]];

      then

        command=$(setup_proxy $command)

      fi

 

      echo $command

    }

command=$(setup_proxy $command) piece looks to be the issue and may need to be switched, but this section of the configuration needs to be reviewed as pointed out this may be the reason why the proxy is not getting properly set in the spawned runner pods.