When gitlab-runner.install=true is set unable to override CI_SERVER_URL environment variable to POST directly to gitlab-nginx-ingress-default-backend container

Summary

When launching the gitlab helm chart to our EKS cluster and using the option to install the gitlab-runner we are unable to tell the gitlab-runner to use a backend url directly to the gitlab server instead of having to go out to the loadbalancer (https://gitlab.mydomain.com).

Steps to reproduce

installing the helm chart and only allow specific IP's access to the gitlab-nginx-ingress-controller LoadBalancer (https://gitlab.mydomain.com):

helm upgrade -i gitlab gitlab/gitlab --timeout 900s \
  --set global.hosts.domain=mydomain.com \
  --set certmanager-issuer.email=certs@mydomain.com \
  --set gitlab-runner.install=true \
  --set gitlab-runner.runners.cloneUrl='gitlab-gitlab-shell.default.svc.cluster.local' \
  --set gitlab-runner.gitlabUrl='gitlab-nginx-ingress-default-backend.default.svc.cluster.local' \
  --set 'nginx-ingress.controller.service.loadBalancerSourceRanges={x.x.x.x/x,x.x.x.x/x}'

More info for above command:

--set gitlab-runner.gitlabUrl - this override does NOT work and does not override CI_SERVER_URL environment variable resulting in gitlab-runner trying to POST to https://gitlab.mydomain.com instead of directly to the gitlab-nginx-ingress-default-backend container.
--set gitlab-runner.runners.cloneUrl - this override works and overrides CLONE_URL environment variable. points directly to gitlab-shell container on our kubernetes cluster.
--set 'nginx-ingress.controller.service.loadBalancerSourceRanges - list of IPs allowed to access gitlab-nginx-ingress-controller LoadBalancer (https://gitlab.mydomain.com)

gitlab-runner logs:

ERROR: Registering runner... failed                 runner=ISb8b4Zh status=couldn't execute POST against https://gitlab.mydomain.com/api/v4/runners: Post https://gitlab.mydomain.com/api/v4/runners: dial tcp 52.73.130.217:443: i/o timeout
PANIC: Failed to register the runner. You may be having network problems. 
Registration attempt 2 of 30

gitlab-runner logs after removing '--set nginx-ingress.controller.service.loadBalancerSourceRanges':

Waiting for stop signal                             builds=0
Metrics server listening                            address=:9252 builds=0
[session_server].listen_address not defined, session endpoints disabled  builds=0
Feeding runners to channel                          builds=0
Starting worker                                     builds=0 worker=0
Starting worker                                     builds=0 worker=9
Starting worker                                     builds=0 worker=2
Starting worker                                     builds=0 worker=1
Starting worker                                     builds=0 worker=4
Starting worker                                     builds=0 worker=3
Starting worker                                     builds=0 worker=6
Starting worker                                     builds=0 worker=5
Starting worker                                     builds=0 worker=8
Starting worker                                     builds=0 worker=7
Dialing: tcp gitlab.mydomain.com:443 ...     
Checking for jobs... nothing                        runner=ioq59jx3

As you can see, this was successful after removing --set nginx-ingress.controller.service.loadBalancerSourceRanges because the gitlab-nginx-ingress-controller LoadBalancer (https://gitlab.mydomain.com) is now allowing all IP addresses to access it. If the --set gitlab-runner.gitlabUrl= would allow access to override CI_SERVER_URL environment variable (which i think is what setting that is suppose to allow) then gitlab-runner could run a POST directly to to the gitlab-nginx-ingress-default-backend container rather than having to go out of the network to https://gitlab.mydomain.com. When running the --set gitlab-runner.runners.cloneUrl it overrides the CLONE_URL in the gitlab-runner container just fine. It only appears to be the --set gitlab-runner.gitlabUrl that is not overriding.

Extra Info That May Help: The _env_vars.tpl i noticed had a couple lines in it that caught my eye.

- name: CI_SERVER_URL
  value: {{ include "gitlab-runner.gitlabUrl" . }}
- name: CLONE_URL
  value: {{ default "" .Values.runners.cloneUrl | quote }}

the CI_SERVER_URL appears to be an 'include' where as the CLONE_URL just reads from Values.

Versions

  • Chart: (tagged version | branch | hash git rev-parse HEAD)
  • Platform:
    • Cloud: EKS
  • Kubernetes: (kubectl version)
    • Client: Client Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.9-eks-4c6976", GitCommit:"4c6976793196d70bc5cd29d56ce5440c9473648e", GitTreeState:"clean", BuildDate:"2020-07-17T19:00:19Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
    • Server: Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.9-eks-d1db3c", GitCommit:"d1db3c46e55f95d6a7d3e5578689371318f95ff9", GitTreeState:"clean", BuildDate:"2020-10-20T22:18:07Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
  • Helm: (helm version) version.BuildInfo{Version:"v3.3.0", GitCommit:"8a4aeec08d67a7b84472007529e8097ec3742105", GitTreeState:"dirty", GoVersion:"go1.14.7"}
Edited by Bryce Kottke