Skip to content

Jobs from later pipelines for a branch are sometimes scheduled before earlier pipelines are completed

Summary

i am running my own GitLab instance and have a single runner and a build CI file with 7 jobs split in 5 stages. At random times, runner seems to grab jobs from subsequent pending builds instead of fully finishing jobs from the current one. Since my builds depend on a 3rd party resource lock, this mixing leads to build failures (ci yaml attached). I can confirm I only have one runner registered in the system (its config is attached below).

I don't know whether this is due to a bug in GitLab or Runner, but this behaviour was present in various versions of 8.13.x, as well as 8.14 of GitLab, and 1.5.2, 1.7.2, and 1.8 of Runner. The reason I know is because I was upgrading in hope of fixing the issue :-) Cancelling & restarting pipeline builds is tiring.

Steps to reproduce

  1. create new project
  2. make sure one runner is available
  3. add .gitlab-ci.yml (attached below)
  4. create a batch of pending jobs (i ran one line change, one commit, one push sequence a few times)
  5. watch pipeline page (see screenshot attached)

Actual behavior

Runner randomly takes jobs from the next pending build instead of finishing the jobs in the current build first

Expected behavior

Runner is expected to finish jobs in the current build first before starting with the next one

Relevant logs and/or screenshots

erratic_runner

**gitlab-ci.yml to demo the problem **

gitlab runner config:

concurrent = 1
check_interval = 5

[[runners]]
  name = "runner"
  url = "https://gitlab.domain.com/"
  token = "t0k3nt0k3nt0k3nt0k3nt0k3n"
  executor = "docker"
  limit = 1
  [runners.docker]
    limit = 1
    tls_verify = false
    image = "ubuntu:16.04"
    privileged = false
    disable_cache = false
    volumes = ["/cache"]
  [runners.cache]

Environment description

docker info
Containers: 6
 Running: 1
 Paused: 0
 Stopped: 5
Images: 8
Server Version: 1.12.3
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 44
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: host overlay null bridge
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-47-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 488.2 MiB
Name: gitlab-runner
ID: ID1:ID1:ID1:ID1:ID1:ID1:ID1:ID1 
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

Used GitLab Runner version

gitlab-runner@gitlab-runner:~$ gitlab-runner --version
Version:      1.8.0
Git revision: 734bc5d
Git branch:   1-8-stable
GO version:   go1.7.3
Built:        Tue, 22 Nov 2016 18:09:06 +0000
OS/Arch:      linux/amd64

running 1.8.0, but 1.7 and 1.5 were also affected.

Using docker executor, but the same behaviour is shown on shell executor as well.