Jobs from later pipelines for a branch are sometimes scheduled before earlier pipelines are completed
Summary
i am running my own GitLab instance and have a single runner and a build CI file with 7 jobs split in 5 stages. At random times, runner seems to grab jobs from subsequent pending builds instead of fully finishing jobs from the current one. Since my builds depend on a 3rd party resource lock, this mixing leads to build failures (ci yaml attached). I can confirm I only have one runner registered in the system (its config is attached below).
I don't know whether this is due to a bug in GitLab or Runner, but this behaviour was present in various versions of 8.13.x, as well as 8.14 of GitLab, and 1.5.2, 1.7.2, and 1.8 of Runner. The reason I know is because I was upgrading in hope of fixing the issue :-) Cancelling & restarting pipeline builds is tiring.
Steps to reproduce
- create new project
- make sure one runner is available
- add .gitlab-ci.yml (attached below)
- create a batch of pending jobs (i ran one line change, one commit, one push sequence a few times)
- watch pipeline page (see screenshot attached)
Actual behavior
Runner randomly takes jobs from the next pending build instead of finishing the jobs in the current build first
Expected behavior
Runner is expected to finish jobs in the current build first before starting with the next one
Relevant logs and/or screenshots
**gitlab-ci.yml to demo the problem **
gitlab runner config:
concurrent = 1
check_interval = 5
[[runners]]
name = "runner"
url = "https://gitlab.domain.com/"
token = "t0k3nt0k3nt0k3nt0k3nt0k3n"
executor = "docker"
limit = 1
[runners.docker]
limit = 1
tls_verify = false
image = "ubuntu:16.04"
privileged = false
disable_cache = false
volumes = ["/cache"]
[runners.cache]
Environment description
docker info
Containers: 6
Running: 1
Paused: 0
Stopped: 5
Images: 8
Server Version: 1.12.3
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 44
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: host overlay null bridge
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-47-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 488.2 MiB
Name: gitlab-runner
ID: ID1:ID1:ID1:ID1:ID1:ID1:ID1:ID1
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
127.0.0.0/8
Used GitLab Runner version
gitlab-runner@gitlab-runner:~$ gitlab-runner --version
Version: 1.8.0
Git revision: 734bc5d
Git branch: 1-8-stable
GO version: go1.7.3
Built: Tue, 22 Nov 2016 18:09:06 +0000
OS/Arch: linux/amd64
running 1.8.0, but 1.7 and 1.5 were also affected.
Using docker executor, but the same behaviour is shown on shell executor as well.