Build failed (system failure): API error (404)

Summary

Builds fail with strange error (file not found).

container_linux.go:247: starting container process caused "exec: "gitlab-runner-build": executable file not found in $PATH" ERROR: Build failed (system failure): API error (404): oci runtime error: container_linux.go:247: starting container process caused "exec: "gitlab-runner-build": executable file not found in $PATH"

Steps to reproduce

Hard to say.

Actual behavior

Build fail with error

Expected behavior

Build without error.

Relevant logs and/or screenshots

In debug mode runner

hecking for builds... received                     build=1837 repo_url=https://XXXXXX/XXXXXXXXXXX/tapisooms.git runner=ba627842
Requeued the runner                                 builds=1 runner=ba627842
Running with gitlab-ci-multi-runner 1.10.4 (b32125f)  build=1837 project=37 runner=ba627842
Shell configuration: environment: []
dockercommand:
- sh
- -c
- "if [ -x /usr/local/bin/bash ]; then\n\texec /usr/local/bin/bash \nelif [ -x /usr/bin/bash
  ]; then\n\texec /usr/bin/bash \nelif [ -x /bin/bash ]; then\n\texec /bin/bash \nelif
  [ -x /usr/local/bin/sh ]; then\n\texec /usr/local/bin/sh \nelif [ -x /usr/bin/sh
  ]; then\n\texec /usr/bin/sh \nelif [ -x /bin/sh ]; then\n\texec /bin/sh \nelse\n\techo
  shell not found\n\texit 1\nfi\n\n"
command: bash
arguments: []
passfile: false
extension: ""
  build=1837 project=37 runner=ba627842
Using Docker executor with image tetraweb/php:7.0 ...  build=1837 project=37 runner=ba627842
Applying docker.Client transport fix: &{false 0xc4201819e0 <nil> 0xc42017e8c0 unix:///var/run/docker.sock 0xc4201c4b80 0xc42004b860 [1 18] [] [] <nil>}  host=unix:///var/run/docker.sock
Creating build volume...                            build=1837 project=37 runner=ba627842
Using container d4c89c2242f7e5615c0980ada3bda61f4c6830b5cf194b49d55a0a7093509041 as cache /builds/XXXXXXXXXXX ...  build=1837 project=37 runner=ba627842
Creating services...                                build=1837 project=37 runner=ba627842
Creating user-defined volumes...                    build=1837 project=37 runner=ba627842
Using container 369c18494c85591d539f841b20fa670cfb296d2d5045f86b25186bcc1d8f1f23 as cache /cache ...  build=1837 project=37 runner=ba627842
Starting Docker command...                          build=1837 project=37 runner=ba627842
Looking for prebuilt image gitlab/gitlab-runner-helper:x86_64-b32125f ...  build=1837 project=37 runner=ba627842
Loading prebuilt image...                           build=1837 project=37 runner=ba627842
Checking for builds... nothing                      runner=ba627842
No credentials found for docker.io                  build=1837 project=37 runner=ba627842
Looking for image sha256:fa7ecddd44448d02a6c7ae3c036435657e0750b4f3e1cf041f65ac03119210c3 ...  build=1837 project=37 runner=ba627842
Removed container runner-ba627842-project-37-concurrent-0-predefined with No such container: runner-ba627842-project-37-concurrent-0-predefined  build=1837 project=37 runner=ba627842
Creating container runner-ba627842-project-37-concurrent-0-predefined ...  build=1837 project=37 runner=ba627842
No credentials found for docker.io                  build=1837 project=37 runner=ba627842
Looking for image tetraweb/php:7.0 ...              build=1837 project=37 runner=ba627842
Pulling docker image tetraweb/php:7.0 ...           build=1837 project=37 runner=ba627842
Submitting build to coordinator... ok               build=1837 runner=ba627842
Appending trace to coordinator... ok                build=1837 build-log=0-181 build-status=running code=202 runner=ba627842 sent-log=0-181 status=202 Accepted
Removed container runner-ba627842-project-37-concurrent-0-build with No such container: runner-ba627842-project-37-concurrent-0-build  build=1837 project=37 runner=ba627842
Creating container runner-ba627842-project-37-concurrent-0-build ...  build=1837 project=37 runner=ba627842
Waiting for signals...                              build=1837 project=37 runner=ba627842
Executing on runner-ba627842-project-37-concurrent-0-predefined the set -eo pipefail
set +o noclobber
: | eval $'echo "Running on $(hostname) via xxxx.XXXXXXXXXXXXXX..."\n'
exit 0
  build=1837 project=37 runner=ba627842
Attaching to container 41222e7c780c564fe0e3c4a87087d52d3509bc2118b78d09b576606feb099927 ...  build=1837 project=37 runner=ba627842
Starting container 41222e7c780c564fe0e3c4a87087d52d3509bc2118b78d09b576606feb099927 ...  build=1837 project=37 runner=ba627842
ERROR: Build failed (system failure): API error (404): oci runtime error: container_linux.go:247: starting container process caused "exec: \"gitlab-runner-build\": executable file not found in $PATH"

  build=1837 project=37 runner=ba627842
Submitting build to coordinator... ok               build=1837 runner=ba627842
Removed container 41222e7c780c564fe0e3c4a87087d52d3509bc2118b78d09b576606feb099927 with <nil>  build=1837 project=37 runner=ba627842
Removed container 5806a861df6cde609bbde29eb30d457e22a75673d23d1646030fd6de9ddd495e with <nil>  build=1837 project=37 runner=ba627842
Closed all idle connections for docker.Client: &{false 0xc420181b00 <nil> 0x2f43a60 unix:///var/run/docker.sock 0xc4201c4b80 0xc42004b860 [1 18] [1 26] [1 18] 0xc420181b60}

Environment description

Gitlab CE from source with docker executors.

docker info
Containers: 3
 Running: 1
 Paused: 0
 Stopped: 2
Images: 5
Server Version: 1.13.1
Storage Driver: overlay
 Backing Filesystem: xfs
 Supports d_type: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-514.2.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.907 GiB
Name: XXXXXXXX
ID:XXXXXXX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Used GitLab Runner version

gitlab-runner -v
Version:      1.10.4
Git revision: b32125f
Git branch:   1-10-stable
GO version:   go1.7.5
Built:        Tue, 31 Jan 2017 14:39:18 +0000
OS/Arch:      linux/amd64