.bash_logout causes clone to fail with "No such file or directory"
Summary
I am currently upgrading all my infrastructure from Debian stretch to buster and while most stuff seems to work just fine, my gitlab-runner just breaks for some strange reason. I did not configure anything beyond apt install gitlab-runner
and gitlab-runner register
with the shell executor. This did work in Debian stretch without any problem. Now, all jobs simply fail.
The jobs seem to fail very early in the setup phase because the runner/executor is failing to create its working directory. The runner's home directory and everything up to /home/gitlab-runner/builds/<hash>/0/<group>/
is present. Permissions are good.
I have tried this with several projects that have a totally different .gitlab-ci.yml
. The only thing that changes it the line number of the error as below in the job log.
Steps to reproduce
Install Gitlab and gitlab-runner on Debian stretch, upgrade to buster, try to run something via the Gitlab CI.
Actual behavior
All jobs fail immediately.
Expected behavior
Jobs should run.
Relevant logs and/or screenshots
Group and project names have been removed and replaced with <group>
and <project>
.
job log
Running with gitlab-runner 12.0.2 (d0b76032)
on local ey8QzqLq
Using Shell executor...
Running on gitlab...
bash: line 82: cd: /home/gitlab-runner/builds/ey8QzqLq/0/<group>/<project>: No such file or directory
ERROR: Job failed: exit status 1
runner log
Jul 09 12:15:36 gitlab gitlab-runner[15338]: Feeding runners to channel builds=0
Jul 09 12:15:37 gitlab gitlab-runner[15338]: Checking for jobs... received job=8036 repo_url=https://<url>/<group>/<project>.git runner=ey8QzqLq
Jul 09 12:15:37 gitlab gitlab-runner[15338]: Checking for jobs... received job=8036 repo_url=https://<url>/<group>/<project>.git runner=ey8QzqLq
Jul 09 12:15:37 gitlab gitlab-runner[15338]: Failed to requeue the runner: builds=1 runner=ey8QzqLq
Jul 09 12:15:37 gitlab gitlab-runner[15338]: Running with gitlab-runner 12.0.2 (d0b76032) job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:37 gitlab gitlab-runner[15338]: on local ey8QzqLq job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:37 gitlab gitlab-runner[15338]: Shell configuration: environment: []
Jul 09 12:15:37 gitlab gitlab-runner[15338]: dockercommand:
Jul 09 12:15:37 gitlab gitlab-runner[15338]: - sh
Jul 09 12:15:37 gitlab gitlab-runner[15338]: - -c
Jul 09 12:15:37 gitlab gitlab-runner[15338]: - "if [ -x /usr/local/bin/bash ]; then\n\texec /usr/local/bin/bash --login\nelif [
Jul 09 12:15:37 gitlab gitlab-runner[15338]: -x /usr/bin/bash ]; then\n\texec /usr/bin/bash --login\nelif [ -x /bin/bash ]; then\n\texec
Jul 09 12:15:37 gitlab gitlab-runner[15338]: /bin/bash --login\nelif [ -x /usr/local/bin/sh ]; then\n\texec /usr/local/bin/sh
Jul 09 12:15:37 gitlab gitlab-runner[15338]: --login\nelif [ -x /usr/bin/sh ]; then\n\texec /usr/bin/sh --login\nelif [ -x /bin/sh
Jul 09 12:15:37 gitlab gitlab-runner[15338]: ]; then\n\texec /bin/sh --login\nelif [ -x /busybox/sh ]; then\n\texec /busybox/sh
Jul 09 12:15:37 gitlab gitlab-runner[15338]: --login\nelse\n\techo shell not found\n\texit 1\nfi\n\n"
Jul 09 12:15:37 gitlab gitlab-runner[15338]: command: su
Jul 09 12:15:37 gitlab gitlab-runner[15338]: arguments:
Jul 09 12:15:37 gitlab gitlab-runner[15338]: - -s
Jul 09 12:15:37 gitlab gitlab-runner[15338]: - /bin/bash
Jul 09 12:15:37 gitlab gitlab-runner[15338]: - gitlab-runner
Jul 09 12:15:37 gitlab gitlab-runner[15338]: - -c
Jul 09 12:15:37 gitlab gitlab-runner[15338]: - bash --login
Jul 09 12:15:37 gitlab gitlab-runner[15338]: passfile: false
Jul 09 12:15:37 gitlab gitlab-runner[15338]: extension: ""
Jul 09 12:15:37 gitlab gitlab-runner[15338]: job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:37 gitlab gitlab-runner[15338]: Using Shell executor... job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:37 gitlab gitlab-runner[15338]: Waiting for signals... job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:37 gitlab gitlab-runner[15338]: Executing build stage build_stage=prepare_script job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:37 gitlab su[15567]: (to gitlab-runner) root on none
Jul 09 12:15:37 gitlab su[15567]: pam_unix(su:session): session opened for user gitlab-runner by (uid=0)
Jul 09 12:15:38 gitlab su[15567]: pam_unix(su:session): session closed for user gitlab-runner
Jul 09 12:15:38 gitlab gitlab-runner[15338]: Executing build stage build_stage=upload_artifacts_on_failure job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:38 gitlab su[15586]: (to gitlab-runner) root on none
Jul 09 12:15:38 gitlab su[15586]: pam_unix(su:session): session opened for user gitlab-runner by (uid=0)
Jul 09 12:15:38 gitlab su[15586]: pam_unix(su:session): session closed for user gitlab-runner
Jul 09 12:15:38 gitlab gitlab-runner[15338]: WARNING: Job failed: exit status 1 duration=138.571383ms job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:38 gitlab gitlab-runner[15338]: WARNING: Job failed: exit status 1 duration=138.571383ms job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:38 gitlab gitlab-runner[15338]: Appending trace to coordinator... ok code=202 job=8036 job-log=0-563 job-status=running runner=ey8QzqLq sent-log=0-562 status=202 Accepted
Jul 09 12:15:38 gitlab gitlab-runner[15338]: Submitting job to coordinator... ok code=200 job=8036 job-status= runner=ey8QzqLq
Jul 09 12:15:38 gitlab gitlab-runner[15338]: ERROR: Failed to process runner builds=0 error=exit status 1 executor=shell runner=ey8QzqLq
Jul 09 12:15:38 gitlab gitlab-runner[15338]: ERROR: Failed to process runner builds=0 error=exit status 1 executor=shell runner=ey8QzqLq
Jul 09 12:15:39 gitlab gitlab-runner[15338]: Feeding runners to channel builds=0
Jul 09 12:15:40 gitlab gitlab-runner[15338]: Checking for jobs... nothing runner=ey8QzqLq
Environment description
I'm using a standard Gitlab omnibus installation on Debian stretch/buster and the gitlab-runner package from packages.gitlab.com/runner/gitlab-runner/debian.
config.toml contents
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "local"
url = "https://<url>/"
token = "<token>"
executor = "shell"
[runners.cache]
[[runners]]
name = "local"
url = "https://<url>/"
token = "<token>"
executor = "shell"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
Used GitLab Runner version
Version: 12.0.2
Git revision: d0b76032
Git branch:
GO version: go1.8.7
Built: 2019-07-07T00:39:59+0000
OS/Arch: linux/amd64
Workaround
As explained in #4449 (comment 189923147) and #4449 (comment 189959431) this is because of the default content of ~/.bash_loguout
below:
if [ "$SHLVL" = 1 ]; then
[ -x /usr/bin/clear_console ] && /usr/bin/clear_console -q
fi
This will try and clear the console and GitLab Runner doesn't like that and fails our bash scripts. A current workaround would be to remove the above from your ~/.bash_logout
file.