Skip to content

Runner doesn't pick up jobs, receives 409 conflict on each attempt.

Summary

Closing this bug - refer to the workaround section.

A singular runner doesn't pick up jobs, receives 409 conflict on each attempt.

gitlab-runner run fails with conflict.

gitlab-runner run-single does pick up job and execute succesfully.

Found #4360 (closed), no advertise_address in config.

Actual behavior

A gitlab-runner 14.7.0 running as a service on ubuntu is hosting a couple of docker runners.

Only one of them is registered with a private project hosted on gitlab.com.

The project has shared runners disabled and has a single job pending. There are no other runners registered with this project.

Since 2022-11-23 runner started logging "409 Conflict" warnings, and is not processing jobs.

log_level = "debug" was added to config.toml but no additional log entries other than: "POST https://gitlab.com/api/v4/jobs/request: 409 Conflict" appear.

Expected behavior

gitlab-runner picks up and executes jobs.

Relevant logs and/or screenshots

A run of gitlab-runner run --debug with settings copied from toml (executor, token, and image) does yield the same result - 409:

Dialing: tcp gitlab.com:443 ...
Feeding runners to channel                          builds=0
WARNING: Checking for jobs... failed                runner=(cut) status=POST https://gitlab.com/api/v4/jobs/request: 409 Conflict

However, a run of gitlab-runner run-single with settings extracted from config.tom did successfully acquire and execute the pending job:

Starting runner for https://gitlab.com with token(cut) ...
Checking for jobs... received                       job=(cut) repo_url=https://gitlab.com/(cut) runner=(cut)

Only one runner is registered with project:

obraz

Runner (cut) specific
Property Name 	Value
Active 	Yes
Protected 	No
Can run untagged jobs 	Yes
Locked to this project 	Yes
Tags 	
Name 	gitlab-runner
Version 	14.7.0
IP Address 	(cut -- correct internet facing IP)
Revision 	98daeee0
Platform 	linux
Architecture 	amd64
Description 	(cut)
Maximum job timeout 	
Last contact 	just now

Environment description

Private runner, registered with a private project hosted on github.com

config.toml contents 
concurrent = 1
check_interval = 0
log_level = "debug"

[session_server]
  listen_address = "[::]:8093"
  session_timeout = 1800

[[runners]]
  name = "(cut)"
  url = "(private)"
  (cut)

[[runners]]
  name = "(cut)"
  url = "(private)"
  (cut)

[[runners]]
  name = "(cut)"
  url = "(private)"
  (cut)

[[runners]]
  name = "(cut)"
  url = "(private)"
  (cut)

[[runners]]
  name = "(cut)"
  url = "https://gitlab.com/"
  token = "77Mm(cut)"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "gitlab/gitlab-runner:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

Used GitLab Runner version

gitlab-runner --version
Version:      14.7.0
Git revision: 98daeee0
Git branch:   14-7-stable
GO version:   go1.17.5
Built:        2022-01-19T17:11:48+0000
OS/Arch:      linux/amd64

Possible fixes

I'd like to investigate further, is there any way to get more logging? The only explanation i have now is that run-single sends a different request than run, that causes the "conflict".

Workarounds

  1. Comment out listen_address under session_sesrver in the Runner config.toml. Please make sure to have the configuration updated before the container is started.
        [session_server]
          # listen_address = "0.0.0.0:8093"
          session_timeout = 1800
  1. If you have set advertise_address in the Runner config.toml to an internal IP, then check that you have enabled the application setting allow_local_requests_from_web_hooks_and_services
Edited by Darren Eastman