Running GitLab Runner on Docker with SELinux fails with permission denied
Summary
tl;dr; we are getting CI_SERVER_TLS_CA_FILE: Permission denied
when we enforce SELinux on the Docker daemon with GitLab Runner itself inside a container and with the Docker executor. However, no SELinux message denying permissions are found in journald.
We run GitLab Runner inside a Docker container (so we launch GitLab Runner as a container). On top of that most of our CI jobs are using the Docker executor. We have been using this set-up for a year and a half successfully (and still do).
However, we want to set the SELinux as a security option for the Docker daemon (running with --selinux-enabled
). SELinux was always in enforce mode on our host (CentOS 7) but we figured out that SELinux was not active for Docker itself.
After activating SELinux, all our containers (the GitLab Runner, but also many other containers) were all up and running, working successfully and no strange log messages in journald.
However, some GitLab CI jobs started to fail and we still haven't figure out what's causing them to fail and why some do work. Our astonishment comes from the following analysis:
- No SELinux error message in journald is reported (e.g. no avc denied type of message), nothing
- For the failed jobs, it fails before our job scripts are even ran, so during what I would call the job initialisation (the internal recipe from GitLab runner):
CI_SERVER_TLS_CA_FILE: Permission denied
- I could not find a discrepancy between failed and success jobs. However it seems that when we use a "service" or are referring "artifacts" from other jobs, this is more likely it will fail.
Once we deactivated SELinux for the Docker daemon, the failed jobs were retried and were then successful.
Steps to reproduce
- Use a host which support SELinux
- Make SELinux enforce
- Install Docker CE 18.06
- Activate
selinux-enabled
(either in daemon.json or as a command line argument) - Check that SELinux is activated:
docker info | grep selinux
should returnselinux
- Create a container for GitLab Runner (see https://docs.gitlab.com/runner/install/docker.html)
- Make sure to follow the instructions on SELinux however we tend to use
:z
rather than:Z
but should work in that scenario equally. - Make sure you register the runner and configure it with the Docker executor.
- Create some pipeline jobs. Try to use artifacts.
- Some jobs should fail
Actual behavior
Some jobs are failing with the following log:
Running with gitlab-runner 10.8.0 (079aad9e)
on Generic Shared Runner 02 cca81505
Using Docker executor with image jruby:9.1.7.0 ...
Pulling docker image jruby:9.1.7.0 ...
Using docker image sha256:2a724b947a786374aab7fbf34f02ff4dbb228d11a9001d9cdfda7f1688d1a0a9 for jruby:9.1.7.0 ...
Running on runner-cca81505-project-238-concurrent-0 via shared-runner02...
/bin/bash: line 6: /builds/gitlab/org/shared/org.common.interfaces.tmp/CI_SERVER_TLS_CA_FILE: Permission denied
ERROR: Job failed: exit code 1
Note that the project name is org.common.interfaces and not org.common.interfaces.tmp.
Expected behavior
Once SElinux is disabled for the Docker daemon only, we get the following log (truncated to show the relevant part):
Running with gitlab-runner 10.8.0 (079aad9e)
on Generic Shared Runner 02 cca81505
Using Docker executor with image jruby:9.1.7.0 ...
Pulling docker image jruby:9.1.7.0 ...
Using docker image sha256:2a724b947a786374aab7fbf34f02ff4dbb228d11a9001d9cdfda7f1688d1a0a9 for jruby:9.1.7.0 ...
Running on runner-cca81505-project-238-concurrent-0 via shared-runner02...
Cloning repository...
Cloning into '/builds/gitlab/org/shared/org.common.interfaces'...
Checking out dced1c34 as master...
$ gem install bundler -v 1.11.2 --no-ri --no-rdoc
Successfully installed bundler-1.11.2
1 gem installed
$ ruby -v
jruby 9.1.7.0 (2.3.1) 2017-01-11 68056ae OpenJDK 64-Bit Server VM 25.121-b13 on 1.8.0_121-8u121-b13-1~bpo8+1-b13 +jit [linux-x86_64]
$ /bin/sh install.sh
...
...
Job succeeded
Relevant logs and/or screenshots
Sadly the error is only reported above. No error in journald. And just a notification of failed job in the GitLab Runner container log:
WARNING: Job failed: exit code 1 job=38863 project=238 runner=cca81505
Environment description
So we are using our own hosted GitLab runners, version 10.8.0 inside a Docker container.
When we activate SELinux we have the following info from Docker:
$ docker info
Containers: 100
Running: 23
Paused: 0
Stopped: 77
Images: 298
Server Version: 18.06.0-ce
Storage Driver: devicemapper
Pool Name: vg_spc-thpl_docker
Pool Blocksize: 524.3kB
Base Device Size: 10.74GB
Backing Filesystem: xfs
Udev Sync Supported: true
Data Space Used: 116.1GB
Data Space Total: 1.44TB
Data Space Available: 1.323TB
Metadata Space Used: 53.73MB
Metadata Space Total: 16.98GB
Metadata Space Available: 16.92GB
Thin Pool Minimum Free Space: 144GB
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.146-RHEL7 (2018-01-22)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d64c661f1d51c48782c9cec8fda7604785f93587
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
seccomp
Profile: default
selinux
Kernel Version: 3.10.0-862.9.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 62.9GiB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
We are running it on bare metal, CentOS 7.5.1804 patched with latest package last weekend.
Used GitLab Runner version
$ docker exec -t shared-runners02 gitlab-runner --version
Version: 10.8.0
Git revision: 079aad9e
Git branch:
GO version: go1.8.7
Built: 2018-05-22T03:24:56+00:00
OS/Arch: linux/amd64