### Summary Runner system failure observed when multiple docker runners are active concurrently. ### What is the current *bug* behavior? Runners work perfectly if called one at a time with no jobs in queue. As soon as there are jobs in a queue, runners start failing till there are no more jobs in queues. Once `number of jobs = number of runners`, all runners finish as expected. ``` Running with gitlab-runner 11.1.0 (081978aa) on docker-runner-2 ff9e524d Using Docker executor with image debian:stretch ... Pulling docker image debian:stretch ... Using docker image sha256:3bbb526d26083e7a65a7a112ed72e1ec58e81384412f2d3fcdbbd87d49fd588d for debian:stretch ... Running on runner-ff9e524d-project-5-concurrent-0 via dfe18fd43fb0... Cloning repository... Cloning into '/builds/robotics/code/beagle_black'... Checking out 80fcbd9b as master... Skipping Git submodules setup $ sh scripts/installation.sh WARNING: apt does not have a stable CLI interface. Use with caution in scripts. Get:2 http://security.debian.org/debian-security stretch/updates InRelease [94.3 kB] Ign:1 http://cdn-fastly.deb.debian.org/debian stretch InRelease Get:3 http://cdn-fastly.deb.debian.org/debian stretch-updates InRelease [91.0 kB] ERROR: Job failed (system failure): Error: No such container: dd70c68ba335678a6fe439f3c8084b08ac013bd9d448b320f34724652a45d768 (executor_docker.go:965:0s) ``` Another log, this time with a previously unseen warning ``` Running with gitlab-runner 11.1.0 (081978aa) on docker-runner-1 77b96ed0 Using Docker executor with image ubuntu:18.04 ... Pulling docker image ubuntu:18.04 ... Using docker image sha256:16508e5c265dcb5c05017a2a8a8228ae12b7b56b2cda0197ed5411bda200a961 for ubuntu:18.04 ... Running on runner-77b96ed0-project-5-concurrent-0 via 77c099be3094... Cloning repository... Cloning into '/builds/robotics/code/beagle_black'... Checking out 80fcbd9b as master... Skipping Git submodules setup WARNING: Possibly zombie container runner-77b96ed0-project-5-concurrent-0-build-4 is disconnected from network gitlab-network $ sh scripts/installation.sh WARNING: apt does not have a stable CLI interface. Use with caution in scripts. Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [83.2 kB] Get:2 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB] ERROR: Job failed (system failure): Error: No such container: 7decabe3805a7d0faf9a51d385cf32d71e91cfc5cbc7b9d1155e314f9d8dd352 (executor_docker.go:965:0s) ``` This error has some more information that can use useful ``` Running with gitlab-runner 11.1.0 (081978aa) on docker-runner-1 77b96ed0 Using Docker executor with image debian:stretch ... Pulling docker image debian:stretch ... Using docker image sha256:3bbb526d26083e7a65a7a112ed72e1ec58e81384412f2d3fcdbbd87d49fd588d for debian:stretch ... ERROR: Job failed (system failure): Error response from daemon: Conflict. The container name "/runner-77b96ed0-project-5-concurrent-0-predefined-0" is already in use by container "97da1d8b016ba3b0bd3ea2d19ed2b42e1e6b06758604c8ef19b41eb3ab3d04fb". You have to remove (or rename) that container to be able to reuse that name. (executor_docker.go:922:0s) ``` Also observed `ERROR: Job failed: exit code 137` as referenced [here (gitlab-com/infrastructure, #2289)](https://gitlab.com/gitlab-com/infrastructure/issues/2289) ### What is the expected *correct* behavior? The pipeline should have gone through without a runner failure. #### Results of GitLab environment info <details> <summary>Expand for output related to GitLab environment info</summary> <pre> System information System: Current User: git Using RVM: no Ruby Version: 2.4.4p296 Gem Version: 2.7.6 Bundler Version:1.16.2 Rake Version: 12.3.1 Redis Version: 3.2.11 Git Version: 2.17.1 Sidekiq Version:5.1.3 Go Version: unknown GitLab information Version: 11.1.4 Revision: 63daf37 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: http://git.custom-domain HTTP Clone URL: http://git.custom-domain/some-group/some-project.git SSH Clone URL: git@git.custom-domain:some-group/some-project.git Using LDAP: no Using Omniauth: no GitLab Shell Version: 7.1.4 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git </pre> </details> #### Results of GitLab application Check <details> <summary>Expand for output related to the GitLab application check</summary> <pre> Checking GitLab Shell ... GitLab Shell version >= 7.1.4 ? ... OK (7.1.4) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 5/1 ... ok 5/2 ... ok 5/4 ... ok 5/5 ... ok 5/6 ... ok 5/7 ... ok 5/8 ... ok 3/10 ... ok 3/11 ... ok 12/12 ... ok 11/13 ... ok 13/14 ... ok 11/15 ... ok 5/16 ... ok 12/17 ... ok 12/19 ... ok 2/20 ... ok 5/21 ... ok 5/22 ... repository is empty Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Sidekiq ... Running? ... yes Number of Sidekiq processes ... 1 Checking Sidekiq ... Finished Reply by email is disabled in config/gitlab.yml Checking LDAP ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab ... Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 5/1 ... yes 5/2 ... yes 5/4 ... yes 5/5 ... yes 5/6 ... yes 5/7 ... yes 5/8 ... yes 3/10 ... yes 3/11 ... yes 12/12 ... yes 11/13 ... yes 13/14 ... yes 11/15 ... yes 5/16 ... yes 12/17 ... yes 12/19 ... yes 2/20 ... yes 5/21 ... yes 5/22 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.4.4) Git version >= 2.9.5 ? ... yes (2.17.1) Git user has default SSH configuration? ... yes Active users: ... 7 Checking GitLab ... Finished </pre> </details>