GitLab CI stuck in pipeline after first stage

Summary

GitLab CI has trouble with continuing after completing the first stage. It will not move to the second stage. Also, the pipeline will stay in the pending stage.

Steps to reproduce

How this can be reproduced is hard to say. I only administrate one GitLab instance with GitLab CI. The issue came up when switching our CI-runner VM. All the old runners are removed.

Expected behaviour

I'd expect for the pipeline to move on after the first stage is completed.

Actual behaviour

The pipeline doesn't move on. It keeps the next stage at created and if there is only one stage the pipeline will stay in the pending stage.

Two example .gitlab-ci.yml files I've tried.

Multiple stages

stages:
- stage1
- stage2
- stage3

stage1:
  stage: stage1
  script:
  - echo "Hello"
  tags:
  - debian

stage2:
  stage: stage2
  script:
  - echo "Hello"
  tags:
  - debian

stage3:
  stage: stage3
  script:
  - echo "Hello"
  tags:
  - debian

Single stage

stage1:
  script:
  - echo "Hello"
  - sleep 60
  tags:
  - debian

Relevant logs and/or screenshots

There is not a lot that stands out. Only this line occurs in syslog at the start of the pipeline. 6cae61a1 is the docker container that is going to be started.

Oct 24 23:56:15 <hostname of CI VM> dockerd[481]: time="2016-10-24T23:56:15.503469175+02:00" level=error msg="Handler for DELETE /v1.18/containers/runner-6cae61a1-project-98-concurrent-0-build returned error: No such container: runner-6cae61a1-project-98-concurrent-0-build"

An API response for a stuck pipeline will look something like this:

{
  "id": 196,
  "sha": "105d7745505155936b4c2dd8d6e9cebd52fc7591",
  "ref": "master",
  "status": "pending",
  "before_sha": "105d7745505155936b4c2dd8d6e9cebd52fc7591",
  "tag": false,
  "yaml_errors": null,
  "user": {
    "name": "Daniel Voogsgerd",
    "username": "daniel",
    "id": 3,
    "state": "active",
    "avatar_url": "https://secure.gravatar.com/avatar/3fb0867f96b63d617a8b4f637e95d05c?s=80&d=identicon",
    "web_url": "https://gitlab.arago.utwente.nl/daniel"
  },
  "created_at": "2016-10-25T06:01:46.000+02:00",
  "updated_at": "2016-10-25T06:01:47.000+02:00",
  "started_at": null,
  "finished_at": null,
  "committed_at": null,
  "duration": null
}

Also two screenshots of the interface where you can see the pipelines being stuck.

Output of checks

Results of GitLab application Check

Checking GitLab Shell ...

GitLab Shell version >= 3.6.6 ? ... OK (3.6.6)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by gitlab:gitlab?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ... 
3/1 ... ok
3/4 ... ok
3/5 ... ok
3/6 ... ok
3/7 ... ok
3/8 ... ok
3/9 ... ok
3/10 ... ok
3/11 ... ok
3/12 ... ok
3/13 ... ok
3/14 ... ok
3/15 ... ok
8/16 ... ok
3/17 ... ok
3/18 ... ok
3/19 ... ok
2/39 ... ok
3/40 ... ok
8/41 ... ok
5/45 ... ok
3/46 ... ok
3/49 ... ok
3/50 ... ok
3/51 ... ok
8/53 ... ok
2/54 ... ok
8/55 ... ok
3/56 ... ok
3/57 ... ok
3/58 ... ok
3/59 ... ok
4/60 ... ok
4/61 ... ok
2/62 ... ok
5/64 ... ok
4/65 ... ok
4/66 ... ok
2/67 ... ok
8/68 ... ok
8/69 ... ok
3/70 ... ok
2/71 ... ok
2/72 ... ok
2/73 ... ok
2/74 ... ok
3/75 ... repository is empty
8/77 ... ok
3/78 ... ok
13/79 ... ok
3/80 ... ok
8/81 ... repository is empty
8/82 ... ok
3/83 ... repository is empty
3/84 ... ok
13/85 ... ok
8/86 ... ok
8/87 ... ok
3/89 ... ok
4/95 ... ok
8/96 ... ok
4/97 ... ok
4/98 ... ok
Running /opt/gitlab/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /home/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Reply by email ...

IMAP server credentials are correct? ... yes
Init.d configured correctly? ... no
  Try fixing it:
  Enable mail_room in the init.d configuration.
  For more information see:
  doc/administration/reply_by_email.md
  Please fix the error above and rerun the checks.
MailRoom running? ... can't check because of previous errors

Checking Reply by email ... Finished

Checking LDAP ...

LDAP users with access to your GitLab server (only showing the first 100 results)
Server: ldapmain

[Redacted: User info]

Checking LDAP ... Finished

Checking GitLab ...

Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... no
  Try fixing it:
  Install the init script
  For more information see:
  doc/install/installation.md in section "Install Init Script"
  Please fix the error above and rerun the checks.
Init script up-to-date? ... can't check because of previous errors
projects have namespace: ... 
3/1 ... yes
3/4 ... yes
3/5 ... yes
3/6 ... yes
3/7 ... yes
3/8 ... yes
3/9 ... yes
3/10 ... yes
3/11 ... yes
3/12 ... yes
3/13 ... yes
3/14 ... yes
3/15 ... yes
8/16 ... yes
3/17 ... yes
3/18 ... yes
3/19 ... yes
2/39 ... yes
3/40 ... yes
8/41 ... yes
5/45 ... yes
3/46 ... yes
3/49 ... yes
3/50 ... yes
3/51 ... yes
8/53 ... yes
2/54 ... yes
8/55 ... yes
3/56 ... yes
3/57 ... yes
3/58 ... yes
3/59 ... yes
4/60 ... yes
4/61 ... yes
2/62 ... yes
5/64 ... yes
4/65 ... yes
4/66 ... yes
2/67 ... yes
8/68 ... yes
8/69 ... yes
3/70 ... yes
2/71 ... yes
2/72 ... yes
2/73 ... yes
2/74 ... yes
3/75 ... yes
8/77 ... yes
3/78 ... yes
13/79 ... yes
3/80 ... yes
8/81 ... yes
8/82 ... yes
3/83 ... yes
3/84 ... yes
13/85 ... yes
8/86 ... yes
8/87 ... yes
3/89 ... yes
4/95 ... yes
8/96 ... yes
4/97 ... yes
4/98 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.1.5)
Your git bin path is "/usr/bin/git"
Git version >= 2.7.3 ? ... no
  Try fixing it:
  Update your git to a version >= 2.7.3 from 2.1.4
  Please fix the error above and rerun the checks.
Active users: 56

Checking GitLab ... Finished

(we will only investigate if the tests are passing)

Sorry not all of them pass, but let me explain ;) :

Init script exists? ... no

We start everything (including mailroom) via systemd. Here is the output of sudo -u gitlab -H ps ux

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
gitlab     301  0.0  0.0  19104  2312 pts/0    R+   03:13   0:00 ps ux
gitlab    7288  0.7  0.1 360428 14600 ?        Ssl  00:13   1:21 /opt/gitlab/gitlab-workhorse/gitlab-workhorse -listenUmask 0 -listenNetwork tcp -listenAddr 127.0.0.1:8181 -authBackend http://127.0.0.1:8080 -authSocket /opt/gitlab/gitlab/tmp/sockets/gitlab.socket -documentRoot /opt/gitlab/gitlab/public -secretPath /opt/gitlab/gitlab/.gitlab_workhorse_secret
gitlab    7307  0.3  0.3 244728 32404 ?        Ssl  00:13   0:36 ruby2.1 /opt/gitlab/gitlab/vendor/bundle/ruby/2.1.0/bin/mail_room -q -c /opt/gitlab/gitlab/config/mail_room.yml
gitlab    7326  0.2  3.7 674684 305056 ?       Sl   00:13   0:25 unicorn_rails master -D -c /opt/gitlab/gitlab/config/unicorn.rb -E production                                                 
gitlab    7337  1.0  4.2 2501160 348336 ?      Sl   00:13   1:49 sidekiq 4.2.1 gitlab [0 of 25 busy]                                                                                                                                                                                                                                                             
gitlab   20588  4.2  4.2 721220 348196 ?       Sl   01:53   3:23 unicorn_rails worker[1] -D -c /opt/gitlab/gitlab/config/unicorn.rb -E production                                              
gitlab   30603  4.1  4.2 726168 349696 ?       Sl   03:05   0:18 unicorn_rails worker[0] -D -c /opt/gitlab/gitlab/config/unicorn.rb -E production

The git version is indeed an old one, but it is the one provided in Debian stable. I've found no evidence linking this issue to git so I hope this isn't an issue.

Results of GitLab environment info

System information
System:		Debian 8.6
Current User:	gitlab
Using RVM:	no
Ruby Version:	2.1.5p273
Gem Version:	2.2.2
Bundler Version:1.7.4
Rake Version:	10.5.0
Sidekiq Version:4.2.1

GitLab information
Version:	8.13.0
Revision:	053a0a2
Directory:	/opt/gitlab/gitlab
DB Adapter:	mysql2
URL:		https://gitlab.arago.utwente.nl
HTTP Clone URL:	https://gitlab.arago.utwente.nl/some-group/some-project.git
SSH Clone URL:	gitlab@gitlab.arago.utwente.nl:some-group/some-project.git
Using LDAP:	yes
Using Omniauth:	no

GitLab Shell
Version:	3.6.6
Repository storage paths:
- default: 	/srv/git/
Hooks:		/opt/gitlab/gitlab-shell/hooks/
Git:		/usr/bin/git

Possible fixes

If only I knew.

Personal note

Apologies for the lack of reproducibility and the slightly different than normal setup. I understand if this makes it hard for you guys to find out what the cause of this all is. I'm ready to help and can provide whatever info is necessary as long as it doesn't contain sensitive data.