Skip to content

gitlab-runner sometimes ignores jobs

Summary

I am having an intermittent issue where a private runner will not pick up a job. Retrying the job will usually kick it off.

Steps to reproduce

Start runner in debug.

gitlab-runner --debug run --config /etc/gitlab-runner/config.toml

Watch the output until you see the following.

Checking for jobs... nothing                        runner=xxxxxxx
Feeding runners to channel                          builds=0

Hit Create Pipeline button immiediatly after seeing the above log message.

Watch the log to see that the jobs are checked but not picked up.

Checking for jobs... nothing                        runner=xxxxxxxx
Feeding runners to channel                          builds=0
Checking for jobs... nothing                        runner=xxxxxxxx
Feeding runners to channel                          builds=0
Checking for jobs... nothing                        runner=xxxxxxxx
Feeding runners to channel                          builds=0

The job is never picked up and stays in the pending state. I have waited more the 45 minutes for it to resolve.

Restarting the runner will cause the job to be picked up. You can also do the following: Open the edit page for the runner. Don't edit the tags but remove a space so there is a difference to save. Save the runner. The jobs is instantly picked up by the runner.

Example Project

https://gitlab.com/jrob007/test

What is the current bug behavior?

The job sits in the pending state.

What is the expected correct behavior?

The job is picked up by the runner.

Relevant logs and/or screenshots

Gitlab runner version

Version:      10.7.0
Git revision: 7c273476
Git branch:   10-7-stable
GO version:   go1.8.7
Built:        2018-04-22T13:43:35+00:00
OS/Arch:      linux/amd64
concurrent = 1
check_interval = 0

[[runners]]
  name = "docker-runer-linux-development"
  url = "https://gitlab.com/"
  token = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  executor = "docker"
  environment = ["HTTPS_PROXY=http://xxxxxxxxxxxxxxxxxxxxxx:1234", "HTTP_PROXY=http://xxxxxxxxxxxxxxxxxxxxxx:1234"]
  pre_clone_script = "git config --global http.proxy $HTTP_PROXY; git config --global https.proxy $HTTPS_PROXY"
  [runners.docker]
    tls_verify = false
    image = "node:latest"
    privileged = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.cache]

Output of checks

This bug happens on GitLab.com

Other relevant information

  • Running behind a proxy.

Things I have tried

  • Waiting.
  • Saving the runner with tags in a different order.
  • Checking Run untagged jobs.

Potential workaround

  • Specifying only the required tags on the runner. Removed tags that were not relevant.