gitlab-ci-multi-runnner isn't closing the sockets to docker?
We use gitlab-ci-multi-runner with docker. Both are installed on the system and communication runs via 127.0.0.1:2376 with TLS authentication. Yesterday all CI jobs started to fail with "Too many files open", the analyze showed that too much sockets were open. It looks like gitlab-ci-multi-runner doesn't close the connection to docker after job is finished (the containers are removed still).
$ netstat -anp | grep ESTAB | grep gitlab-ci
tcp 0 0 127.0.0.1:60089 127.0.0.1:2376 ESTABLISHED 19157/gitlab-ci-mul
tcp 0 0 127.0.0.1:60166 127.0.0.1:2376 ESTABLISHED 19157/gitlab-ci-mul
tcp 0 0 127.0.0.1:60033 127.0.0.1:2376 ESTABLISHED 19157/gitlab-ci-mul
# Now I trigger some single&simple CI job and wait for it
# and now the stats after the run (docker shows no running containers)
$ netstat -anp | grep ESTAB | grep gitlab-ci
tcp 0 0 127.0.0.1:60089 127.0.0.1:2376 ESTABLISHED 19157/gitlab-ci-mul
tcp 0 0 127.0.0.1:60498 127.0.0.1:2376 ESTABLISHED 19157/gitlab-ci-mul
tcp 0 0 127.0.0.1:60166 127.0.0.1:2376 ESTABLISHED 19157/gitlab-ci-mul
tcp 0 0 127.0.0.1:60033 127.0.0.1:2376 ESTABLISHED 19157/gitlab-ci-mul
I did the upgrade to the latest available version of gitlab-ci-multi-runner and still see the same behavior like above.
Following versions and environment are used:
$ cat /etc/*release*
NAME="Red Hat Enterprise Linux Server"
VERSION="7.1 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="7.1"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.1 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.1:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.1
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.1"
Red Hat Enterprise Linux Server release 7.1 (Maipo)
Red Hat Enterprise Linux Server release 7.1 (Maipo)
cpe:/o:redhat:enterprise_linux:7.1:ga:server
$ rpm -qa | grep gitlab
gitlab-ci-multi-runner-0.5.5_1_g69bc934-1.x86_64
$ docker --version
Docker version 1.7.1, build 3043001/1.7.1
Following configuration file is used:
concurrent = 10
[[runners]]
name = "centos-chefdk"
url = "<URL>"
token = "<TOKEN>"
limit = 10
executor = "docker"
builds_dir = ""
clean_environment = true
shell = ""
disable_verbose = false
[runners.docker]
host = "tcp://127.0.0.1:2376"
tls_cert_path = "/home/gitlab_ci_multi_runner/.docker"
hostname = ""
image = "centos-chefdk"
privileged = false
disable_cache = true
disable_pull = true
volumes = []
cache_dir = ""
registry = ""
[[runners]]
name = "centos-chefdk-vagrant"
url = "<URL>"
token = "<TOKEN>"
limit = 1
executor = "docker"
builds_dir = ""
clean_environment = true
shell = ""
disable_verbose = false
[runners.docker]
host = "tcp://127.0.0.1:2376"
tls_cert_path = "/home/gitlab_ci_multi_runner/.docker"
hostname = ""
image = "centos-chefdk-vagrant"
privileged = false
disable_cache = true
disable_pull = true
volumes = []
cache_dir = ""
registry = ""