Runner doesn't pick up jobs, receives 409 conflict on each attempt.
Summary
Closing this bug - refer to the workaround section.
A singular runner doesn't pick up jobs, receives 409 conflict on each attempt.
gitlab-runner run
fails with conflict.
gitlab-runner run-single
does pick up job and execute succesfully.
Found #4360 (closed), no advertise_address
in config.
Actual behavior
A gitlab-runner 14.7.0 running as a service on ubuntu is hosting a couple of docker runners.
Only one of them is registered with a private project hosted on gitlab.com.
The project has shared runners disabled and has a single job pending. There are no other runners registered with this project.
Since 2022-11-23 runner started logging "409 Conflict" warnings, and is not processing jobs.
log_level = "debug"
was added to config.toml but no additional log entries other than:
"POST https://gitlab.com/api/v4/jobs/request: 409 Conflict"
appear.
Expected behavior
gitlab-runner picks up and executes jobs.
Relevant logs and/or screenshots
A run of gitlab-runner run --debug
with settings copied from toml (executor, token, and image) does yield the same result - 409:
Dialing: tcp gitlab.com:443 ...
Feeding runners to channel builds=0
WARNING: Checking for jobs... failed runner=(cut) status=POST https://gitlab.com/api/v4/jobs/request: 409 Conflict
However, a run of gitlab-runner run-single
with settings extracted from config.tom did successfully acquire and execute the pending job:
Starting runner for https://gitlab.com with token(cut) ...
Checking for jobs... received job=(cut) repo_url=https://gitlab.com/(cut) runner=(cut)
Only one runner is registered with project:
Runner (cut) specific
Property Name Value
Active Yes
Protected No
Can run untagged jobs Yes
Locked to this project Yes
Tags
Name gitlab-runner
Version 14.7.0
IP Address (cut -- correct internet facing IP)
Revision 98daeee0
Platform linux
Architecture amd64
Description (cut)
Maximum job timeout
Last contact just now
Environment description
Private runner, registered with a private project hosted on github.com
config.toml contents
concurrent = 1
check_interval = 0
log_level = "debug"
[session_server]
listen_address = "[::]:8093"
session_timeout = 1800
[[runners]]
name = "(cut)"
url = "(private)"
(cut)
[[runners]]
name = "(cut)"
url = "(private)"
(cut)
[[runners]]
name = "(cut)"
url = "(private)"
(cut)
[[runners]]
name = "(cut)"
url = "(private)"
(cut)
[[runners]]
name = "(cut)"
url = "https://gitlab.com/"
token = "77Mm(cut)"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "gitlab/gitlab-runner:latest"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
Used GitLab Runner version
gitlab-runner --version
Version: 14.7.0
Git revision: 98daeee0
Git branch: 14-7-stable
GO version: go1.17.5
Built: 2022-01-19T17:11:48+0000
OS/Arch: linux/amd64
Possible fixes
I'd like to investigate further, is there any way to get more logging?
The only explanation i have now is that run-single
sends a different request than run
, that causes the "conflict".
Workarounds
- Comment out
listen_address
undersession_sesrver
in the Runnerconfig.toml
. Please make sure to have the configuration updated before the container is started.
[session_server]
# listen_address = "0.0.0.0:8093"
session_timeout = 1800
- If you have set
advertise_address
in the Runner config.toml to an internal IP, then check that you have enabled the application settingallow_local_requests_from_web_hooks_and_services