gitlab-runner sometimes ignores jobs
Summary
I am having an intermittent issue where a private runner will not pick up a job. Retrying the job will usually kick it off.
Steps to reproduce
Start runner in debug.
gitlab-runner --debug run --config /etc/gitlab-runner/config.toml
Watch the output until you see the following.
Checking for jobs... nothing runner=xxxxxxx
Feeding runners to channel builds=0
Hit Create Pipeline
button immiediatly after seeing the above log message.
Watch the log to see that the jobs are checked but not picked up.
Checking for jobs... nothing runner=xxxxxxxx
Feeding runners to channel builds=0
Checking for jobs... nothing runner=xxxxxxxx
Feeding runners to channel builds=0
Checking for jobs... nothing runner=xxxxxxxx
Feeding runners to channel builds=0
The job is never picked up and stays in the pending state. I have waited more the 45 minutes for it to resolve.
Restarting the runner will cause the job to be picked up. You can also do the following: Open the edit page for the runner. Don't edit the tags but remove a space so there is a difference to save. Save the runner. The jobs is instantly picked up by the runner.
Example Project
https://gitlab.com/jrob007/test
What is the current bug behavior?
The job sits in the pending state.
What is the expected correct behavior?
The job is picked up by the runner.
Relevant logs and/or screenshots
Gitlab runner version
Version: 10.7.0
Git revision: 7c273476
Git branch: 10-7-stable
GO version: go1.8.7
Built: 2018-04-22T13:43:35+00:00
OS/Arch: linux/amd64
concurrent = 1
check_interval = 0
[[runners]]
name = "docker-runer-linux-development"
url = "https://gitlab.com/"
token = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
executor = "docker"
environment = ["HTTPS_PROXY=http://xxxxxxxxxxxxxxxxxxxxxx:1234", "HTTP_PROXY=http://xxxxxxxxxxxxxxxxxxxxxx:1234"]
pre_clone_script = "git config --global http.proxy $HTTP_PROXY; git config --global https.proxy $HTTPS_PROXY"
[runners.docker]
tls_verify = false
image = "node:latest"
privileged = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
Output of checks
This bug happens on GitLab.com
Other relevant information
- Running behind a proxy.
Things I have tried
- Waiting.
- Saving the runner with tags in a different order.
- Checking
Run untagged jobs
.
Potential workaround
- Specifying only the required tags on the runner. Removed tags that were not relevant.