Executor fails to fetch project on retried job builds
Summary
Any time if I try to retry a job from the gitlab admin, I get this:
Running with gitlab-runner 12.5.0 (577f813d)
on exampleserver-bash fce2c0de
Using Shell executor...
Running on exampleserver...
Fetching changes with git depth set to 50...
Bestehendes Git-Repository in /path/to/gitlab-runner/builds/fce2c0de/0/gitlab.path.deleted/by/me/.git/ neuinitialisiert
remote: You are not allowed to download code from this project.
fatal: unable to access 'https://gitlab-ci-token:[MASKED]@gitlab.path.deleted/by/me.git/': The requested URL returned error: 403
ERROR: Job failed: exit status 1
If happens at least with the shell executor, and at least with the debian stretch version of the gitlab-ce, downloaded by apt from the gitlab.com repository.
It happens exclusively for the jobs, whose retry was initiated from the gitlab project administration (project admin page -> "CI / CD" -> "jobs" -> "retry" icons in the last column).
Steps to reproduce
Try to retry any job, at least with the shell executor.
My .gitlab-ci.yml
is very simple:
deploy:
when: always
script: "bash ./onpush.sh"
Although I don't think that it would change anything - the runner already fails as it tries to fetch the project, thus it does not reach the point where it could read the .gitlab-ci.yml
file.
Actual behavior
Gitlab-runner can't clone the project, it gets 403.
It happens exclusively for retried builds (independently that the original build was successful or not). For the builds triggered by new pushes, it does not happen.
Expected behavior
Registered gitlab-runners should be able to clone to projects where they are assigned, if the job was retried or original, it should not matter.
Relevant logs and/or screenshots
There is a relevant log in the description.
Environment description
Both gitlab-ce and gitlab-runner are coming from the upstream, binary, debian stretch repository ( https://packages.gitlab.com/gitlab/gitlab-ce/debian/ ).
These are specific runners, assigned to the project manually.
My config.toml
of the runner is this:
concurrent = 1
check_interval = 0
log_level = "debug"
[session_server]
session_timeout = 1800
[[runners]]
name = "unrelated-docker-runner"
...unrelated-part-removed...
[[runners]]
name = "exampleserver-bash"
url = "https://gitlab.example.com"
token = "secret-token-removed"
executor = "shell"
limit = 1
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
Used GitLab Runner version
My gitlab-runner --version
output is:
Version: 12.5.0
Git revision: 577f813d
Git branch: 12-5-stable
GO version: go1.10.8
Built: 2019-11-20T09:14:54+0000
OS/Arch: linux/amd64
Possible fixes
As far I know, that "MASKED" gitlab-ci-token -s are re-generated on every job execution. I suspect, it might not happen on retried builds. I suspect, it might be some version incompatibility between gitlab (gitaly?) and the gitlab-runner. If my assumtion is correct, please update this gitlab-runner. Currently I can't retry builds.