Skip to content

gitlab-ci-multi-runnner isn't closing the sockets to docker?

We use gitlab-ci-multi-runner with docker. Both are installed on the system and communication runs via 127.0.0.1:2376 with TLS authentication. Yesterday all CI jobs started to fail with "Too many files open", the analyze showed that too much sockets were open. It looks like gitlab-ci-multi-runner doesn't close the connection to docker after job is finished (the containers are removed still).

$ netstat -anp | grep ESTAB | grep gitlab-ci
tcp        0      0 127.0.0.1:60089         127.0.0.1:2376          ESTABLISHED 19157/gitlab-ci-mul 
tcp        0      0 127.0.0.1:60166         127.0.0.1:2376          ESTABLISHED 19157/gitlab-ci-mul 
tcp        0      0 127.0.0.1:60033         127.0.0.1:2376          ESTABLISHED 19157/gitlab-ci-mul
# Now I trigger some single&simple CI job and wait for it
# and now the stats after the run (docker shows no running containers)
$ netstat -anp | grep ESTAB | grep gitlab-ci
tcp        0      0 127.0.0.1:60089         127.0.0.1:2376          ESTABLISHED 19157/gitlab-ci-mul 
tcp        0      0 127.0.0.1:60498         127.0.0.1:2376          ESTABLISHED 19157/gitlab-ci-mul 
tcp        0      0 127.0.0.1:60166         127.0.0.1:2376          ESTABLISHED 19157/gitlab-ci-mul 
tcp        0      0 127.0.0.1:60033         127.0.0.1:2376          ESTABLISHED 19157/gitlab-ci-mul

I did the upgrade to the latest available version of gitlab-ci-multi-runner and still see the same behavior like above.

Following versions and environment are used:

$ cat /etc/*release*
NAME="Red Hat Enterprise Linux Server"
VERSION="7.1 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="7.1"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.1 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.1:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.1
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.1"
Red Hat Enterprise Linux Server release 7.1 (Maipo)
Red Hat Enterprise Linux Server release 7.1 (Maipo)
cpe:/o:redhat:enterprise_linux:7.1:ga:server
$ rpm -qa | grep gitlab
gitlab-ci-multi-runner-0.5.5_1_g69bc934-1.x86_64
$ docker --version
Docker version 1.7.1, build 3043001/1.7.1

Following configuration file is used:

concurrent = 10

[[runners]]
  name = "centos-chefdk"
  url = "<URL>"
  token = "<TOKEN>"
  limit = 10
  executor = "docker"
  builds_dir = ""
  clean_environment = true
  shell = ""
  disable_verbose = false
  [runners.docker]
    host = "tcp://127.0.0.1:2376"
    tls_cert_path = "/home/gitlab_ci_multi_runner/.docker"
    hostname = ""
    image = "centos-chefdk"
    privileged = false
    disable_cache = true
    disable_pull = true
    volumes = []
    cache_dir = ""
    registry = ""

[[runners]]
  name = "centos-chefdk-vagrant"
  url = "<URL>"
  token = "<TOKEN>"
  limit = 1
  executor = "docker"
  builds_dir = ""
  clean_environment = true
  shell = ""
  disable_verbose = false
  [runners.docker]
    host = "tcp://127.0.0.1:2376"
    tls_cert_path = "/home/gitlab_ci_multi_runner/.docker"
    hostname = ""
    image = "centos-chefdk-vagrant"
    privileged = false
    disable_cache = true
    disable_pull = true
    volumes = []
    cache_dir = ""
    registry = ""