Kubernetes runner helm chart ignores some configuration
Summary
We've installed Gitlab CI Kubernetes runners though Helm (chart 0.53.2, runner version 16.0.2) and has mirrored all the required images (we don't allow running external images).
After updating the update, Gitlab Runners were started in the default namespace and using gitlab.com-hosted images.
Steps to reproduce
Relevant bits of values.yaml:
8 image:
9 registry: harbor.example.com
10 image: mirror/registry.gitlab.com/gitlab-org/gitlab-runner
11 tag: ubuntu-v16.0.2
85 runners:
86 namespace: "systems-gitlab-ci"
87
88 # Don't run jobs that aren't tagged in gitlab-ci
89 runUntagged: false
...
92 helpers:
93 image: "harbor.example.com/standard-images/gitlab-runner-helper:v16.0.2"
94 cpuLimit: "500m"
95 cpuRequests: "250m"
96 memoryLimit: "500Mi"
97 memoryRequests: "250Mi"
114 config: |
115 # Runners is -partially- overwritten: https://gitlab.com/gitlab-org/gitlab-runner/-/blob/main/docs/register/index.md#runners-configuration-template -file
116 [[runners]]
117 executor="kubernetes"
118
119 # "error: could not lock config file //.gitconfig: Permission denied". Wonder if setting builds_dir will help?
120 # See <https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3511>
121 builds_dir="/builds"
122 environment=["HOME=/builds"]
123
131 [runners.kuberentes]
132 # This bit doesn't seem to be used at all!?
133 namespace = "{{ .Release.Namespace }}"
134 namespace_overwrite_allowed = ""
135 # Set default image?
136 image = "harbor.example.com/standard-images/ci/ubuntu:focal-rootless"
137 priority_class_name = "sheddable"
138 host = "https://kubernetes.default.svc:6443"
139 privileged = false
140 pull_policy = "if-not-present"
141 poll_interval = 1
142 poll_timeout = 600
143 # Size of our workers - per docs, this is the way to go, but it's currently ignored runtime?!
144 cpu_request = "{{ .Values.runners.builds.cpuRequests }}"
145 cpu_limit = "{{ .Values.runners.builds.cpuLimit }}"
146 memory_request = "{{ .Values.runners.builds.memoryRequests }}"
147 memory_limit = "{{ .Values.runners.builds.memoryLimit }}"
148
149 # Again, we should only set this per docs, but it has no effect!?
150 helper_image = "{{ .Values.runners.helpers.image }}"It turned out that a whole bunch of environment variables, previously given to the runner, has been removed from the helm chart.
Actual behavior
Attempts to start pods in default-namespace (instead of the expected systems-gitlab-ci). And pulls an external helper image, which our admission controller promptly rejects.
Expected behavior
It should start a pods/jobs in the systems-gitlab-ci-namespace with our self-hosted images.
To make things work as a start, I had to add
envVars:
- name: KUBERNETES_NAMESPACE
value: systems-gitlab-ci
- name: KUBERNETES_HELPER_IMAGE
value: "harbor.example.com/standard-images/gitlab-runner-helper:v16.0.2"Environment description
config.toml contents
# Runners is -partially- overwritten: https://gitlab.com/gitlab-org/gitlab-runner/-/blob/main/docs/register/index.md#runners-configuration-template-file
[[runners]]
executor="kubernetes"
# "error: could not lock config file //.gitconfig: Permission denied". Wonder if setting builds_dir will help?
# See <https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3511>
builds_dir="/builds"
environment=["HOME=/builds"]
[runners.cache]
Type = "s3"
Path = "gitlab-ci/kubernetes-runners"
Shared = true
[runners.cache.s3]
ServerAddress = "s3.example.com"
BucketName = "ocsysinfra-gitlab-k8s-runner-cache"
[runners.kuberentes]
# This bit doesn't seem to be used at all!?
namespace = "{{ .Release.Namespace }}"
namespace_overwrite_allowed = ""
# Set default image?
image = "harbor.example.com/standard-images/ci/ubuntu:focal-rootless"
priority_class_name = "sheddable"
host = "https://kubernetes.default.svc:6443"
privileged = false
pull_policy = "if-not-present"
poll_interval = 1
poll_timeout = 600
# Size of our workers - per docs, this is the way to go, but it's currently ignored runtime?!
cpu_request = "{{ .Values.runners.builds.cpuRequests }}"
cpu_limit = "{{ .Values.runners.builds.cpuLimit }}"
memory_request = "{{ .Values.runners.builds.memoryRequests }}"
memory_limit = "{{ .Values.runners.builds.memoryLimit }}"
# Again, we should only set this per docs, but it has no effect!?
helper_image = "{{ .Values.runners.helpers.image }}"
helper_cpu_request = "{{ .Values.runners.helpers.cpuRequests }}"
helper_cpu_limit = "{{ .Values.runners.helpers.cpuLimit }}"
helper_memory_request = "{{ .Values.runners.helpers.memoryRequests }}"
helper_memory_limit = "{{ .Values.runners.helpers.memoryLimit }}"
service_account_overwrite_allowed = ""
pod_annotations_overwrite_allowed = ""
[runners.kubernetes.pod_labels]
"ci_system" = "gitlab-ci"
"job_id" = "${CI_JOB_ID}"
"job_name" = "${CI_JOB_NAME}"
"pipeline_id" = "${CI_PIPELINE_IID}"
"project" = "${CI_PROJECT_PATH}"
"runner_name" = "{{ .Values.runners.name }}"
[[runners.kubernetes.volumes.empty_dir]]
name = "cache"
mount_path = "/cache"
# Ask k8s to spread CI jobs across all nodes as much as possible
[[runners.kubernetes.affinity.pod_anti_affinity.preferred_during_scheduling_ignored_during_execution]]
weight = 100
[runners.kubernetes.affinity.pod_anti_affinity.preferred_during_scheduling_ignored_during_execution.pod_affinity_term]
topology_key = "kubernetes.io/hostname"
namespaces = ["{{ .Release.Namespace }}"]
[runners.kubernetes.affinity.pod_anti_affinity.preferred_during_scheduling_ignored_during_execution.pod_affinity_term.label_selector]
[[runners.kubernetes.affinity.pod_anti_affinity.preferred_during_scheduling_ignored_during_execution.pod_affinity_term.label_selector.match_expressions]]
key = "ci_system"
operator = "In"
values = ["gitlab-ci"]Used GitLab Runner version
16.0.2, installed through helm-chart 0.53.2
Possible fixes
envVars:
- name: KUBERNETES_NAMESPACE
value: systems-gitlab-ci
- name: KUBERNETES_HELPER_IMAGE
value: "harbor.example.com/standard-images/gitlab-runner-helper:v16.0.2"