GitLab Runner Helm chart does not bind Docker socket
Summary
As far as I can tell, the Helm chart for the GitLab runner does not contain the proper volume mount (https://gitlab.com/charts/gitlab-runner/blob/master/templates/deployment.yaml) for /var/run/docker.sock
, despite making the option available in the documentation:
runners:
## Run all containers with the privileged flag enabled
## This will allow the docker:dind image to run if you need to run Docker
## commands. Please read the docs before turning this on:
## ref: https://docs.gitlab.com/runner/executors/kubernetes.html#using-docker-dind
##
privileged: true
Steps to reproduce
With a Kubernetes cluster, install the GitLab runner Helm chart with something like the following values:
$ helm upgrade \
--debug \
--force \
--install \
--namespace gitlab-runner \
--set runners.privileged='true' \
--timeout 600 \
--values values.yaml \
--version 0.7.0-rc1 \
gitlab-runner \
gitlab/gitlab-runner
Make sure the values.yaml
file container your registration token to connect to GitLab.
Actual behavior
The agent will register itself, the pods within Kubernetes will spin up correctly, but running any job afterwards that uses the docker
command will result in an error like this:
$ docker build -t mycompany/mycompany-app:$CI_BUILD_REF_NAME .
time="2019-07-10T08:58:49Z" level=error msg="failed to dial gRPC: cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory"
context canceled
Expected behavior
Docker commands should work because the pod has been set to privileged=true
.
Relevant logs and/or screenshots
N/A
Environment description
GitLab runner: gitlab/gitlab-runner:alpine-v11.11.4 GitLab Runner Helm chart version: gitlab-runner-0.6.1
Contents of `helm get gitlab-runner` chart in my cluster
---
# Source: gitlab-runner/templates/secrets.yaml
apiVersion: v1
kind: Secret
metadata:
name: "gitlab-runner-gitlab-runner"
labels:
app: gitlab-runner-gitlab-runner
chart: gitlab-runner-0.6.1
release: "gitlab-runner"
heritage: "Tiller"
type: Opaque
data:
runner-registration-token: "<my-token>"
runner-token: ""
---
# Source: gitlab-runner/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: gitlab-runner-gitlab-runner
labels:
app: gitlab-runner-gitlab-runner
chart: gitlab-runner-0.6.1
release: "gitlab-runner"
heritage: "Tiller"
data:
entrypoint: |
#!/bin/bash
set -e
mkdir -p /home/gitlab-runner/.gitlab-runner/
cp /scripts/config.toml /home/gitlab-runner/.gitlab-runner/
# Register the runner
if [[ -f /secrets/accesskey && -f /secrets/secretkey ]]; then
export CACHE_S3_ACCESS_KEY=$(cat /secrets/accesskey)
export CACHE_S3_SECRET_KEY=$(cat /secrets/secretkey)
fi
if [[ -f /secrets/gcs-applicaton-credentials-file ]]; then
export GOOGLE_APPLICATION_CREDENTIALS="/secrets/gcs-applicaton-credentials-file"
else
if [[ -f /secrets/gcs-access-id && -f /secrets/gcs-private-key ]]; then
export CACHE_GCS_ACCESS_ID=$(cat /secrets/gcs-access-id)
# echo -e used to make private key multiline (in google json auth key private key is oneline with \n)
export CACHE_GCS_PRIVATE_KEY=$(echo -e $(cat /secrets/gcs-private-key))
fi
fi
if [[ -f /secrets/runner-registration-token ]]; then
export REGISTRATION_TOKEN=$(cat /secrets/runner-registration-token)
fi
if [[ -f /secrets/runner-token ]]; then
export CI_SERVER_TOKEN=$(cat /secrets/runner-token)
fi
if ! sh /scripts/register-the-runner; then
exit 1
fi
# Start the runner
/entrypoint run --user=gitlab-runner \
--working-directory=/home/gitlab-runner
config.toml: |
concurrent = 20
check_interval = 30
log_level = "info"
listen_address = '[::]:9252'
configure: |
set -e
cp /init-secrets/* /secrets
register-the-runner: |
#!/bin/bash
MAX_REGISTER_ATTEMPTS=30
for i in $(seq 1 "${MAX_REGISTER_ATTEMPTS}"); do
echo "Registration attempt ${i} of ${MAX_REGISTER_ATTEMPTS}"
/entrypoint register \
--non-interactive
retval=$?
if [ ${retval} = 0 ]; then
break
elif [ ${i} = ${MAX_REGISTER_ATTEMPTS} ]; then
exit 1
fi
sleep 5
done
exit 0
check-live: |
#!/bin/bash
if /usr/bin/pgrep -f .*register-the-runner; then
exit 0
elif /usr/bin/pgrep gitlab.*runner; then
exit 0
else
exit 1
fi
---
# Source: gitlab-runner/templates/service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: gitlab-runner-gitlab-runner
labels:
app: gitlab-runner-gitlab-runner
chart: gitlab-runner-0.6.1
release: "gitlab-runner"
heritage: "Tiller"
---
# Source: gitlab-runner/templates/role.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: "Role"
metadata:
name: gitlab-runner-gitlab-runner
labels:
app: gitlab-runner-gitlab-runner
chart: gitlab-runner-0.6.1
release: "gitlab-runner"
heritage: "Tiller"
rules:
- apiGroups: [""]
resources: ["*"]
verbs: ["*"]
---
# Source: gitlab-runner/templates/role-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: "RoleBinding"
metadata:
name: gitlab-runner-gitlab-runner
labels:
app: gitlab-runner-gitlab-runner
chart: gitlab-runner-0.6.1
release: "gitlab-runner"
heritage: "Tiller"
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: "Role"
name: gitlab-runner-gitlab-runner
subjects:
- kind: ServiceAccount
name: gitlab-runner-gitlab-runner
namespace: "gitlab-runner"
---
# Source: gitlab-runner/templates/deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: gitlab-runner-gitlab-runner
labels:
app: gitlab-runner-gitlab-runner
chart: gitlab-runner-0.6.1
release: "gitlab-runner"
heritage: "Tiller"
spec:
replicas: 1
template:
metadata:
labels:
app: gitlab-runner-gitlab-runner
annotations:
checksum/configmap: c178206fb8c47686accc9ffec43f2884667c6b42137402cd51dbfe8006e3377c
checksum/secrets: 35aa7b6d2e632956847a073d67a1268be10350d460aee3f55eb0b3ba958ba91a
prometheus.io/scrape: 'true'
prometheus.io/port: '9252'
iam.amazonaws.com/role: "ToolingRoleGitLabRunner"
spec:
securityContext:
runAsUser: 100
fsGroup: 65533
initContainers:
- name: configure
command: ['sh', '/config/configure']
image: gitlab/gitlab-runner:alpine-v11.11.4
env:
- name: CI_SERVER_URL
value: "https://gitlab.com/"
- name: CLONE_URL
value: ""
- name: RUNNER_REQUEST_CONCURRENCY
value: "5"
- name: RUNNER_EXECUTOR
value: "kubernetes"
- name: REGISTER_LOCKED
value: "true"
- name: RUNNER_TAG_LIST
value: "k8s"
- name: KUBERNETES_IMAGE
value: "ubuntu:18.04"
- name: KUBERNETES_PRIVILEGED
value: "true"
- name: KUBERNETES_NAMESPACE
value: "gitlab-runner"
- name: KUBERNETES_CPU_LIMIT
value: ""
- name: KUBERNETES_MEMORY_LIMIT
value: ""
- name: KUBERNETES_CPU_REQUEST
value: ""
- name: KUBERNETES_MEMORY_REQUEST
value: ""
- name: KUBERNETES_SERVICE_ACCOUNT
value: ""
- name: KUBERNETES_SERVICE_CPU_LIMIT
value: ""
- name: KUBERNETES_SERVICE_MEMORY_LIMIT
value: ""
- name: KUBERNETES_SERVICE_CPU_REQUEST
value: ""
- name: KUBERNETES_SERVICE_MEMORY_REQUEST
value: ""
- name: KUBERNETES_HELPER_CPU_LIMIT
value: ""
- name: KUBERNETES_HELPER_MEMORY_LIMIT
value: ""
- name: KUBERNETES_HELPER_CPU_REQUEST
value: ""
- name: KUBERNETES_HELPER_MEMORY_REQUEST
value: ""
- name: KUBERNETES_HELPER_IMAGE
value: ""
- name: KUBERNETES_PULL_POLICY
value: ""
volumeMounts:
- name: runner-secrets
mountPath: /secrets
readOnly: false
- name: scripts
mountPath: /config
readOnly: true
- name: init-runner-secrets
mountPath: /init-secrets
readOnly: true
resources:
{}
serviceAccountName: gitlab-runner-gitlab-runner
containers:
- name: gitlab-runner-gitlab-runner
image: gitlab/gitlab-runner:alpine-v11.11.4
lifecycle:
preStop:
exec:
command: ["gitlab-runner", "unregister", "--all-runners"]
imagePullPolicy: "Always"
command: ["/bin/bash", "/scripts/entrypoint"]
env:
- name: CI_SERVER_URL
value: "https://gitlab.com/"
- name: CLONE_URL
value: ""
- name: RUNNER_REQUEST_CONCURRENCY
value: "5"
- name: RUNNER_EXECUTOR
value: "kubernetes"
- name: REGISTER_LOCKED
value: "true"
- name: RUNNER_TAG_LIST
value: "k8s"
- name: KUBERNETES_IMAGE
value: "ubuntu:18.04"
- name: KUBERNETES_PRIVILEGED
value: "true"
- name: KUBERNETES_NAMESPACE
value: "gitlab-runner"
- name: KUBERNETES_CPU_LIMIT
value: ""
- name: KUBERNETES_MEMORY_LIMIT
value: ""
- name: KUBERNETES_CPU_REQUEST
value: ""
- name: KUBERNETES_MEMORY_REQUEST
value: ""
- name: KUBERNETES_SERVICE_ACCOUNT
value: ""
- name: KUBERNETES_SERVICE_CPU_LIMIT
value: ""
- name: KUBERNETES_SERVICE_MEMORY_LIMIT
value: ""
- name: KUBERNETES_SERVICE_CPU_REQUEST
value: ""
- name: KUBERNETES_SERVICE_MEMORY_REQUEST
value: ""
- name: KUBERNETES_HELPER_CPU_LIMIT
value: ""
- name: KUBERNETES_HELPER_MEMORY_LIMIT
value: ""
- name: KUBERNETES_HELPER_CPU_REQUEST
value: ""
- name: KUBERNETES_HELPER_MEMORY_REQUEST
value: ""
- name: KUBERNETES_HELPER_IMAGE
value: ""
- name: KUBERNETES_PULL_POLICY
value: ""
livenessProbe:
exec:
command: ["/bin/bash", "/scripts/check-live"]
initialDelaySeconds: 60
timeoutSeconds: 1
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
readinessProbe:
exec:
command: ["/usr/bin/pgrep","gitlab.*runner"]
initialDelaySeconds: 10
timeoutSeconds: 1
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
ports:
- name: metrics
containerPort: 9252
volumeMounts:
- name: runner-secrets
mountPath: /secrets
- name: etc-gitlab-runner
mountPath: /home/gitlab-runner/.gitlab-runner
- name: scripts
mountPath: /scripts
resources:
{}
volumes:
- name: runner-secrets
emptyDir:
medium: "Memory"
- name: etc-gitlab-runner
emptyDir:
medium: "Memory"
- name: init-runner-secrets
projected:
sources:
- secret:
name: "gitlab-runner-gitlab-runner"
items:
- key: runner-registration-token
path: runner-registration-token
- key: runner-token
path: runner-token
- name: scripts
configMap:
name: gitlab-runner-gitlab-runner
Used GitLab Runner version
$ gitlab-runner --version
Version: 11.11.4
Git revision: e828d3bc
Git branch:
GO version: go1.8.7
Built: 2019-07-07T00:29:25+0000
OS/Arch: linux/amd64
Possible fixes
Unless I'm mistaken, and totally missed something here, I think the Deployment
YAML just needs something like this if privileged
is set to true
:
...
securityContext:
privileged: true
volumeMounts:
- mountPath: /var/run/docker.sock
name: docker-socket
volumes:
- name: docker-socket
hostPath:
path: /var/run/docker.sock
...