.bash_logout causes clone to fail with "No such file or directory" (#4449) · Issues · GitLab.org / gitlab-runner

.bash_logout causes clone to fail with "No such file or directory"

Summary

I am currently upgrading all my infrastructure from Debian stretch to buster and while most stuff seems to work just fine, my gitlab-runner just breaks for some strange reason. I did not configure anything beyond apt install gitlab-runner and gitlab-runner register with the shell executor. This did work in Debian stretch without any problem. Now, all jobs simply fail.

The jobs seem to fail very early in the setup phase because the runner/executor is failing to create its working directory. The runner's home directory and everything up to /home/gitlab-runner/builds/<hash>/0/<group>/ is present. Permissions are good.

I have tried this with several projects that have a totally different .gitlab-ci.yml. The only thing that changes it the line number of the error as below in the job log.

Steps to reproduce

Install Gitlab and gitlab-runner on Debian stretch, upgrade to buster, try to run something via the Gitlab CI.

Actual behavior

All jobs fail immediately.

Expected behavior

Jobs should run.

Relevant logs and/or screenshots

Group and project names have been removed and replaced with <group> and <project>.

job log

Running with gitlab-runner 12.0.2 (d0b76032)
  on local ey8QzqLq
Using Shell executor...
Running on gitlab...
bash: line 82: cd: /home/gitlab-runner/builds/ey8QzqLq/0/<group>/<project>: No such file or directory
ERROR: Job failed: exit status 1

runner log

Jul 09 12:15:36 gitlab gitlab-runner[15338]: Feeding runners to channel                          builds=0
Jul 09 12:15:37 gitlab gitlab-runner[15338]: Checking for jobs... received                       job=8036 repo_url=https://<url>/<group>/<project>.git runner=ey8QzqLq
Jul 09 12:15:37 gitlab gitlab-runner[15338]: Checking for jobs... received                       job=8036 repo_url=https://<url>/<group>/<project>.git runner=ey8QzqLq
Jul 09 12:15:37 gitlab gitlab-runner[15338]: Failed to requeue the runner:                       builds=1 runner=ey8QzqLq
Jul 09 12:15:37 gitlab gitlab-runner[15338]: Running with gitlab-runner 12.0.2 (d0b76032)        job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:37 gitlab gitlab-runner[15338]:   on local ey8QzqLq                                 job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:37 gitlab gitlab-runner[15338]: Shell configuration: environment: []
Jul 09 12:15:37 gitlab gitlab-runner[15338]: dockercommand:
Jul 09 12:15:37 gitlab gitlab-runner[15338]: - sh
Jul 09 12:15:37 gitlab gitlab-runner[15338]: - -c
Jul 09 12:15:37 gitlab gitlab-runner[15338]: - "if [ -x /usr/local/bin/bash ]; then\n\texec /usr/local/bin/bash --login\nelif [
Jul 09 12:15:37 gitlab gitlab-runner[15338]:   -x /usr/bin/bash ]; then\n\texec /usr/bin/bash --login\nelif [ -x /bin/bash ]; then\n\texec
Jul 09 12:15:37 gitlab gitlab-runner[15338]:   /bin/bash --login\nelif [ -x /usr/local/bin/sh ]; then\n\texec /usr/local/bin/sh
Jul 09 12:15:37 gitlab gitlab-runner[15338]:   --login\nelif [ -x /usr/bin/sh ]; then\n\texec /usr/bin/sh --login\nelif [ -x /bin/sh
Jul 09 12:15:37 gitlab gitlab-runner[15338]:   ]; then\n\texec /bin/sh --login\nelif [ -x /busybox/sh ]; then\n\texec /busybox/sh
Jul 09 12:15:37 gitlab gitlab-runner[15338]:   --login\nelse\n\techo shell not found\n\texit 1\nfi\n\n"
Jul 09 12:15:37 gitlab gitlab-runner[15338]: command: su
Jul 09 12:15:37 gitlab gitlab-runner[15338]: arguments:
Jul 09 12:15:37 gitlab gitlab-runner[15338]: - -s
Jul 09 12:15:37 gitlab gitlab-runner[15338]: - /bin/bash
Jul 09 12:15:37 gitlab gitlab-runner[15338]: - gitlab-runner
Jul 09 12:15:37 gitlab gitlab-runner[15338]: - -c
Jul 09 12:15:37 gitlab gitlab-runner[15338]: - bash --login
Jul 09 12:15:37 gitlab gitlab-runner[15338]: passfile: false
Jul 09 12:15:37 gitlab gitlab-runner[15338]: extension: ""
Jul 09 12:15:37 gitlab gitlab-runner[15338]:   job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:37 gitlab gitlab-runner[15338]: Using Shell executor...                             job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:37 gitlab gitlab-runner[15338]: Waiting for signals...                              job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:37 gitlab gitlab-runner[15338]: Executing build stage                               build_stage=prepare_script job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:37 gitlab su[15567]: (to gitlab-runner) root on none
Jul 09 12:15:37 gitlab su[15567]: pam_unix(su:session): session opened for user gitlab-runner by (uid=0)
Jul 09 12:15:38 gitlab su[15567]: pam_unix(su:session): session closed for user gitlab-runner
Jul 09 12:15:38 gitlab gitlab-runner[15338]: Executing build stage                               build_stage=upload_artifacts_on_failure job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:38 gitlab su[15586]: (to gitlab-runner) root on none
Jul 09 12:15:38 gitlab su[15586]: pam_unix(su:session): session opened for user gitlab-runner by (uid=0)
Jul 09 12:15:38 gitlab su[15586]: pam_unix(su:session): session closed for user gitlab-runner
Jul 09 12:15:38 gitlab gitlab-runner[15338]: WARNING: Job failed: exit status 1                  duration=138.571383ms job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:38 gitlab gitlab-runner[15338]: WARNING: Job failed: exit status 1                  duration=138.571383ms job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:38 gitlab gitlab-runner[15338]: Appending trace to coordinator... ok                code=202 job=8036 job-log=0-563 job-status=running runner=ey8QzqLq sent-log=0-562 status=202 Accepted
Jul 09 12:15:38 gitlab gitlab-runner[15338]: Submitting job to coordinator... ok                 code=200 job=8036 job-status= runner=ey8QzqLq
Jul 09 12:15:38 gitlab gitlab-runner[15338]: ERROR: Failed to process runner                     builds=0 error=exit status 1 executor=shell runner=ey8QzqLq
Jul 09 12:15:38 gitlab gitlab-runner[15338]: ERROR: Failed to process runner                     builds=0 error=exit status 1 executor=shell runner=ey8QzqLq
Jul 09 12:15:39 gitlab gitlab-runner[15338]: Feeding runners to channel                          builds=0
Jul 09 12:15:40 gitlab gitlab-runner[15338]: Checking for jobs... nothing                        runner=ey8QzqLq

Environment description

I'm using a standard Gitlab omnibus installation on Debian stretch/buster and the gitlab-runner package from packages.gitlab.com/runner/gitlab-runner/debian.

config.toml contents

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "local"
  url = "https://<url>/"
  token = "<token>"
  executor = "shell"
  [runners.cache]

[[runners]]
  name = "local"
  url = "https://<url>/"
  token = "<token>"
  executor = "shell"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

Used GitLab Runner version

Version:      12.0.2
Git revision: d0b76032
Git branch:   
GO version:   go1.8.7
Built:        2019-07-07T00:39:59+0000
OS/Arch:      linux/amd64

Workaround

As explained in #4449 (comment 189923147) and #4449 (comment 189959431) this is because of the default content of ~/.bash_loguout below:

if [ "$SHLVL" = 1 ]; then
    [ -x /usr/bin/clear_console ] && /usr/bin/clear_console -q
fi

This will try and clear the console and GitLab Runner doesn't like that and fails our bash scripts. A current workaround would be to remove the above from your ~/.bash_logout file.

Duplicated

#4474 (closed)

## Summary

I am currently upgrading all my infrastructure from Debian stretch to buster and while most stuff seems to work just fine, my gitlab-runner just breaks for some strange reason. I did not configure anything beyond `apt install gitlab-runner` and `gitlab-runner register` with the shell executor. This did work in Debian stretch without any problem. Now, all jobs simply fail.

The jobs seem to fail very early in the setup phase because the runner/executor is failing to create its working directory.  The runner's home directory and everything up to `/home/gitlab-runner/builds/<hash>/0/<group>/` is present. Permissions are good.

I have tried this with several projects that have a totally different <code>.gitlab-ci.yml</code>. The only thing that changes it the line number of the error as below in the job log.

## Steps to reproduce

Install Gitlab and gitlab-runner on Debian stretch, upgrade to buster, try to run something via the Gitlab CI.

## Actual behavior

All jobs fail immediately.

## Expected behavior

Jobs should run.

## Relevant logs and/or screenshots

Group and project names have been removed and replaced with `<group>` and `<project>`.

```
Running with gitlab-runner 12.0.2 (d0b76032)
  on local ey8QzqLq
Using Shell executor...
Running on gitlab...
bash: line 82: cd: /home/gitlab-runner/builds/ey8QzqLq/0/<group>/<project>: No such file or directory
ERROR: Job failed: exit status 1
```
</details>

<details>
<summary> runner log </summary>

```
Jul 09 12:15:36 gitlab gitlab-runner[15338]: Feeding runners to channel                          builds=0
Jul 09 12:15:37 gitlab gitlab-runner[15338]: Checking for jobs... received                       job=8036 repo_url=https://<url>/<group>/<project>.git runner=ey8QzqLq
Jul 09 12:15:37 gitlab gitlab-runner[15338]: Checking for jobs... received                       job=8036 repo_url=https://<url>/<group>/<project>.git runner=ey8QzqLq
Jul 09 12:15:37 gitlab gitlab-runner[15338]: Failed to requeue the runner:                       builds=1 runner=ey8QzqLq
Jul 09 12:15:37 gitlab gitlab-runner[15338]: Running with gitlab-runner 12.0.2 (d0b76032)        job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:37 gitlab gitlab-runner[15338]:   on local ey8QzqLq                                 job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:37 gitlab gitlab-runner[15338]: Shell configuration: environment: []
Jul 09 12:15:37 gitlab gitlab-runner[15338]: dockercommand:
Jul 09 12:15:37 gitlab gitlab-runner[15338]: - sh
Jul 09 12:15:37 gitlab gitlab-runner[15338]: - -c
Jul 09 12:15:37 gitlab gitlab-runner[15338]: - "if [ -x /usr/local/bin/bash ]; then\n\texec /usr/local/bin/bash --login\nelif [
Jul 09 12:15:37 gitlab gitlab-runner[15338]:   -x /usr/bin/bash ]; then\n\texec /usr/bin/bash --login\nelif [ -x /bin/bash ]; then\n\texec
Jul 09 12:15:37 gitlab gitlab-runner[15338]:   /bin/bash --login\nelif [ -x /usr/local/bin/sh ]; then\n\texec /usr/local/bin/sh
Jul 09 12:15:37 gitlab gitlab-runner[15338]:   --login\nelif [ -x /usr/bin/sh ]; then\n\texec /usr/bin/sh --login\nelif [ -x /bin/sh
Jul 09 12:15:37 gitlab gitlab-runner[15338]:   ]; then\n\texec /bin/sh --login\nelif [ -x /busybox/sh ]; then\n\texec /busybox/sh
Jul 09 12:15:37 gitlab gitlab-runner[15338]:   --login\nelse\n\techo shell not found\n\texit 1\nfi\n\n"
Jul 09 12:15:37 gitlab gitlab-runner[15338]: command: su
Jul 09 12:15:37 gitlab gitlab-runner[15338]: arguments:
Jul 09 12:15:37 gitlab gitlab-runner[15338]: - -s
Jul 09 12:15:37 gitlab gitlab-runner[15338]: - /bin/bash
Jul 09 12:15:37 gitlab gitlab-runner[15338]: - gitlab-runner
Jul 09 12:15:37 gitlab gitlab-runner[15338]: - -c
Jul 09 12:15:37 gitlab gitlab-runner[15338]: - bash --login
Jul 09 12:15:37 gitlab gitlab-runner[15338]: passfile: false
Jul 09 12:15:37 gitlab gitlab-runner[15338]: extension: ""
Jul 09 12:15:37 gitlab gitlab-runner[15338]:   job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:37 gitlab gitlab-runner[15338]: Using Shell executor...                             job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:37 gitlab gitlab-runner[15338]: Waiting for signals...                              job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:37 gitlab gitlab-runner[15338]: Executing build stage                               build_stage=prepare_script job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:37 gitlab su[15567]: (to gitlab-runner) root on none
Jul 09 12:15:37 gitlab su[15567]: pam_unix(su:session): session opened for user gitlab-runner by (uid=0)
Jul 09 12:15:38 gitlab su[15567]: pam_unix(su:session): session closed for user gitlab-runner
Jul 09 12:15:38 gitlab gitlab-runner[15338]: Executing build stage                               build_stage=upload_artifacts_on_failure job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:38 gitlab su[15586]: (to gitlab-runner) root on none
Jul 09 12:15:38 gitlab su[15586]: pam_unix(su:session): session opened for user gitlab-runner by (uid=0)
Jul 09 12:15:38 gitlab su[15586]: pam_unix(su:session): session closed for user gitlab-runner
Jul 09 12:15:38 gitlab gitlab-runner[15338]: WARNING: Job failed: exit status 1                  duration=138.571383ms job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:38 gitlab gitlab-runner[15338]: WARNING: Job failed: exit status 1                  duration=138.571383ms job=8036 project=23 runner=ey8QzqLq
Jul 09 12:15:38 gitlab gitlab-runner[15338]: Appending trace to coordinator... ok                code=202 job=8036 job-log=0-563 job-status=running runner=ey8QzqLq sent-log=0-562 status=202 Accepted
Jul 09 12:15:38 gitlab gitlab-runner[15338]: Submitting job to coordinator... ok                 code=200 job=8036 job-status= runner=ey8QzqLq
Jul 09 12:15:38 gitlab gitlab-runner[15338]: ERROR: Failed to process runner                     builds=0 error=exit status 1 executor=shell runner=ey8QzqLq
Jul 09 12:15:38 gitlab gitlab-runner[15338]: ERROR: Failed to process runner                     builds=0 error=exit status 1 executor=shell runner=ey8QzqLq
Jul 09 12:15:39 gitlab gitlab-runner[15338]: Feeding runners to channel                          builds=0
Jul 09 12:15:40 gitlab gitlab-runner[15338]: Checking for jobs... nothing                        runner=ey8QzqLq
```
</details>

## Environment description

I'm using a standard Gitlab omnibus installation on Debian stretch/buster and the gitlab-runner package from packages.gitlab.com/runner/gitlab-runner/debian.

<details>
<summary> config.toml contents </summary>

```toml
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "local"
  url = "https://<url>/"
  token = "<token>"
  executor = "shell"
  [runners.cache]

[[runners]]
  name = "local"
  url = "https://<url>/"
  token = "<token>"
  executor = "shell"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

```
</details>

### Used GitLab Runner version

```
Version:      12.0.2
Git revision: d0b76032
Git branch:   
GO version:   go1.8.7
Built:        2019-07-07T00:39:59+0000
OS/Arch:      linux/amd64
```

## Workaround

As explained in https://gitlab.com/gitlab-org/gitlab-runner/issues/4449#note_189923147 and https://gitlab.com/gitlab-org/gitlab-runner/issues/4449#note_189959431 this is because of the default content of `~/.bash_loguout` below:

```
if [ "$SHLVL" = 1 ]; then
    [ -x /usr/bin/clear_console ] && /usr/bin/clear_console -q
fi
```

This will try and clear the console and GitLab Runner doesn't like that and fails our bash scripts. A current workaround would be to remove the above from your `~/.bash_logout` file.

## Duplicated

- https://gitlab.com/gitlab-org/gitlab-runner/issues/4474

Edited Oct 02, 2019 by Steve Azzopardi