## Summary The `get_sources` stage fails intermittently and silently, causing CI jobs to fail with no error output. The root cause is in the `IfGitVersionIsAtLeast` function in `shells/bash.go`: it uses a `sort ... | head -n1` pipeline to compare git versions. `head -n1` closes the pipe after reading the first line, so `sort` receives SIGPIPE (exit code 141) when writing the second line. Because the generated script enables `set -o pipefail` + `set -o errexit`, and the whole script is wrapped in `: | eval` (which normalizes the exit code to 1), this SIGPIPE silently aborts the entire get_sources script — the clone/fetch commands after the version check never execute, the project directory is never created, and the next stage fails with `No such file or directory` when it tries to `cd` into it, ultimately resulting in `Job failed: exit code 1`. The issue is intermittent because whether SIGPIPE occurs depends on kernel scheduling timing and pipe buffer state. Retries usually succeed. Trigger conditions: `FF_USE_GIT_NATIVE_CLONE=true` + `GIT_STRATEGY=clone` (both lead to the `IfGitVersionIsAtLeast` version-check branch). Both can be set as CI/CD variables (in `.gitlab-ci.yml` `variables`, project-level variables, or runner environment) — modifying the runner's config.toml is not required. This is independent of the git version in the helper image — regardless of whether the git version is above or below the threshold `2.49`, the version-check code itself executes and can trigger SIGPIPE. **Affected versions**: the `head -n1` pipeline in `IfGitVersionIsAtLeast` has never been modified; the current `main` branch ([`shells/bash.go:304`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/main/shells/bash.go?ref_type=heads#L304)) still contains the same code. Therefore **all gitlab-runner versions are affected**, not just 18.11.3. ## Steps to reproduce > **About the executor**: The root cause of this bug is in the bash script generated by `shells/bash.go`, and is independent of the executor type. As long as the runner uses the bash shell generator (which the vast majority of executors do) and the feature flag + git strategy conditions below are met, the bug is triggered — it is not limited to docker-autoscaler; any executor such as docker / kubernetes / shell can reproduce it. Two reproduction methods are provided: Method 1 uses a plain shell script for quick reproduction (no GitLab/runner environment needed); Method 2 triggers pipelines in a loop on a real GitLab instance. ### Method 1: plain shell script (quick, no GitLab environment needed) This script replicates the `Finish()` structure from `bash.go` (`set -o pipefail` + `set -o errexit` + `: | eval` wrapper) plus the version-check pipeline from `IfGitVersionIsAtLeast`. Run it on any machine with bash. > **Why 10000 lines of data**: the real `IfGitVersionIsAtLeast` only feeds 2 lines to `sort` (`required_ver` and `current_ver`). On bare metal, 2 lines is too few — `sort` usually finishes writing before `head` closes the pipe, so SIGPIPE does not fire. Inside a container (different scheduling timing), `sort` is more likely to write the second line after `head` has closed, triggering SIGPIPE intermittently — this is exactly the production failure scenario. To make the script reproduce reliably in any environment, 10000 lines are used here to force `sort` to still be writing when `head` closes, thus reliably triggering SIGPIPE. <details> <summary> minimal shell reproducer script </summary> ```bash #!/usr/bin/env bash # Minimal reproducer for the SIGPIPE issue in IfGitVersionIsAtLeast (bash.go:274-281). # Mimics the script structure generated by bash.go Finish() (bash.go:407, 411-413): # set -o pipefail + set -o errexit + ": | eval" wrapper set -u ITERATIONS="${1:-50}" TRUNCATOR="${2:-head}" # "head" (original, bash.go:277) or "sed" (fix) # Inner script: replicates the IfGitVersionIsAtLeast version-check pipeline. # Uses 10000 lines instead of the real 2 to force sort to still be writing # when head closes, reliably triggering SIGPIPE. if [ "$TRUNCATOR" = "head" ]; then inner='minimum_ver="$(seq 1 10000 | sort -n | head -n1)" echo "SCRIPT_COMPLETED"' else inner='minimum_ver="$(seq 1 10000 | sort -n | sed -n '"'"'1p'"'"')" echo "SCRIPT_COMPLETED"' fi # Outer structure: replicates bash.go:407, 411-413 wrapper="if set -o | grep pipefail > /dev/null; then set -o pipefail; fi; set -o errexit set +o noclobber : | eval $(printf '%q' "$inner") exit 0" pass=0 fail=0 for ((i=1; i<=ITERATIONS; i++)); do out="$(printf '%s\n' "$wrapper" | bash 2>&1)" rc=$? if [ "$rc" -eq 0 ]; then pass=$((pass+1)) else fail=$((fail+1)) fi done echo "truncator=${TRUNCATOR} iterations=${ITERATIONS}" echo " pass=${pass} fail=${fail}" echo " (fail = SIGPIPE(141) -> errexit -> normalized to exit 1 by ': | eval' wrapper)" ``` Run: ```sh # Original version (head -n1), expect failures bash reproducer.sh 50 head # Fixed version (sed -n '1p'), expect 0 failures bash reproducer.sh 50 sed ``` Tested results: - `head` (original): 49 out of 50 failed - `sed` (fix): all 50 succeeded </details> ### Method 2: real pipeline loop 1. Configure a runner with any executor (no need to modify the runner's config.toml). 2. The git version in the helper image can be anything (in this case git 2.47.3, below the threshold 2.49; but versions above 2.49 also trigger it, because the version-check code executes either way). 3. Set `FF_USE_GIT_NATIVE_CLONE=true` + `GIT_STRATEGY=clone` via CI/CD variables in `.gitlab-ci.yml` (see example below), then trigger any job. - Note: `FF_USE_GIT_NATIVE_CLONE` takes effect as a job variable only if the runner's config.toml does not explicitly configure this flag in `[runners.feature_flags]` (runner config takes precedence over job variables). If the runner already configures this flag, the runner config wins. Since this issue is **intermittent** (depends on kernel scheduling timing), a single trigger may not hit it. It is recommended to use a script that triggers pipelines in a loop to quickly catch a failure. > About the trigger interval `INTERVAL=20`: the script waits for the current pipeline to finish (polling until success/failed/canceled/skipped) before sleeping `INTERVAL` seconds and triggering the next one, so pipelines do not pile up. `INTERVAL` is the gap between two pipelines, not the trigger frequency — it must be set so the runner can keep up. If your jobs take longer to run, increase it accordingly; for a quick reproducer job like `echo "hello"` that finishes in seconds, 20s is sufficient. <details> <summary> trigger pipeline loop script (generic, replace placeholders) </summary> Replace the following placeholders before use: - `GITLAB_URL` — your GitLab instance URL - `PROJECT` — your project path (URL-encoded, e.g. `group%2Fproject`) - `REF` — the branch/tag to trigger - `GITLAB_TOKEN` — environment variable, an access token with permission to trigger pipelines ```bash #!/usr/bin/env bash # Trigger a pipeline every INTERVAL seconds, poll its status, and stop (printing # the failed job links) as soon as any job fails. # Used to quickly reproduce intermittent issues. set -u GITLAB_URL="https://your-gitlab.example.com" # <- replace with your GitLab URL PROJECT="group%2Fproject" # <- replace with your project (URL-encoded) PROJECT_API="${GITLAB_URL}/api/v4/projects/${PROJECT}" REF="master" INTERVAL=20 # gap between pipelines (seconds) POLL_INTERVAL=10 # poll interval for a single pipeline status (seconds) POLL_TIMEOUT=600 # wait timeout for a single pipeline (seconds) : "${GITLAB_TOKEN:?please set the GITLAB_TOKEN environment variable first}" i=0 while true; do i=$((i+1)) ts=$(date '+%Y-%m-%d %H:%M:%S') resp=$(curl -s --max-time 15 --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \ -X POST "${PROJECT_API}/pipeline?ref=${REF}" 2>&1) pid=$(echo "$resp" | python3 -c "import sys,json; print(json.load(sys.stdin).get('id',''))" 2>/dev/null) if [ -z "$pid" ]; then echo "[${ts}] #${i} trigger failed: ${resp}" sleep "$INTERVAL" continue fi echo "[${ts}] #${i} triggered pipeline=${pid}, waiting..." elapsed=0 status="" while [ "$elapsed" -lt "$POLL_TIMEOUT" ]; do status=$(curl -s --max-time 15 --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \ "${PROJECT_API}/pipelines/${pid}" \ | python3 -c "import sys,json; print(json.load(sys.stdin).get('status',''))" 2>/dev/null) case "$status" in success|failed|canceled|skipped) break ;; esac sleep "$POLL_INTERVAL" elapsed=$((elapsed + POLL_INTERVAL)) done done_ts=$(date '+%Y-%m-%d %H:%M:%S') if [ "$status" != "success" ]; then echo "[${done_ts}] #${i} pipeline=${pid} status=${status}, fetching jobs..." jobs=$(curl -s --max-time 15 --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \ "${PROJECT_API}/pipelines/${pid}/jobs") failed_jobs=$(echo "$jobs" | python3 -c " import sys,json jobs=json.load(sys.stdin) for j in jobs: if j.get('status') in ('failed','canceled'): print(f\" job={j.get('id')} name={j.get('name')} status={j.get('status')}\\n {j.get('web_url')}\") " 2>/dev/null) echo "[${done_ts}] #${i} FAILED, stopping script." echo "failed jobs:" echo "$failed_jobs" exit 1 fi echo "[${done_ts}] #${i} pipeline=${pid} success, next trigger in ${INTERVAL}s." sleep "$INTERVAL" done ``` </details> <details> <summary> .gitlab-ci.yml </summary> ```yml # This bug is independent of the specific job definition; any job can reproduce it. # The key conditions are set via CI/CD variables, no need to modify runner config.toml: variables: FF_USE_GIT_NATIVE_CLONE: "true" GIT_STRATEGY: "clone" example-job: script: - echo "hello" ``` </details> ## Actual behavior The job fails silently during the `get_sources` stage: 1. After outputting `Gitaly correlation ID: ...`, there is no clone/fetch-related output. 2. The get_sources stage ends immediately and jumps to `upload_artifacts_on_failure`. 3. The next stage fails with `No such file or directory` when executing `cd /builds/<project>` (the project directory was never created). 4. Ultimately `Job failed: exit code 1`, with no error message anywhere indicating the root cause. With `CI_DEBUG_TRACE=true` enabled, the `set -x` trace reveals the exact abort point: after the version-check pipeline `sort ... | head -n1` completes and `minimum_ver` is assigned, the immediately following `if [ "$minimum_ver" = "$required_ver" ]; then` statement never executes — the script aborts right there. Retries usually succeed (due to the non-determinism of SIGPIPE). ## Expected behavior The version-check pipeline in `IfGitVersionIsAtLeast` should not abort script execution due to SIGPIPE. Regardless of whether the git version meets the threshold, the version check should complete normally, followed by the clone or fallback fetch commands, and the job should pull code as usual. ## Relevant logs and/or screenshots The following is a `set -x` trace excerpt from a failed job on our self-managed GitLab instance (with `CI_DEBUG_TRACE=true` enabled), showing the exact abort point: <details> <summary> job log (key excerpt) </summary> ```sh # 1. The version-check command generated by the script (with head -n1) + minimum_ver="$(printf '%s\n%s' "$required_ver" "$current_ver" | sort -t '.' -k 1,1n -k 2,2n -k 3,3n | head -n1)" + if [ "$minimum_ver" = "$required_ver" ]; then # 2. set -x trace expands the pipeline internals; printf / sort / head -n1 all executed +++ printf '%s\n%s' 2.49 2.47.3 +++ sort -t . -k 1,1n -k 2,2n -k 3,3n +++ head -n1 ++ minimum_ver=2.47.3 # <- assignment completed # 3. But the immediately following if statement never appears in the trace! # The script aborts silently here (get_sources stage ends directly) section_end:1785750571:get_sources section_start:1785750571:upload_artifacts_on_failure Uploading artifacts for failed job # 4. The next stage fails to cd into the project directory (never created) /bin/bash: line 240: cd: /builds/group/project: No such file or directory # 5. Final failure, no root-cause information ERROR: Job failed: exit code 1 ``` </details> Abort mechanism (code-level): - `shells/bash.go:277` — `head -n1` truncates the `sort` pipeline; `sort` receives SIGPIPE (exit code 141) when writing the second line. - `shells/bash.go:407` — the script enables `set -o pipefail` + `set -o errexit`; the pipeline exit code 141 triggers errexit. - `shells/bash.go:411-413` — the `: | eval $'...'` wrapper normalizes the exit code to 1. - `helpers/featureflags/flags.go:135-141` — `FF_ENABLE_BASH_EXIT_CODE_CHECK` defaults to false, so errexit exits without printing an error message → silent. ## Environment description - Self-managed GitLab instance + self-managed runner (not GitLab.com shared runner). - Executor: this case uses docker-autoscaler (helper container runs in Docker). **But the executor type is not a required condition** — the root cause is in the bash script generator; any executor can reproduce it. - Helper image: digest `571952e6`, with git `2.47.3`. - `FF_USE_GIT_NATIVE_CLONE=true` and `GIT_STRATEGY=clone` are both set via **project-level CI/CD variables** (GitLab project Settings → CI/CD → Variables); the runner's config.toml does not configure these. - Note: `FF_USE_GIT_NATIVE_CLONE` takes effect as a job variable only if the runner's config.toml does not explicitly configure this flag in `[runners.feature_flags]` (runner config takes precedence over job variables, source `common/build_settings.go:250-265`). In this case config.toml does not configure it, so the project-level variable takes effect. <details> <summary> config.toml contents </summary> ```toml # FF_USE_GIT_NATIVE_CLONE and GIT_STRATEGY are NOT configured in config.toml; # they are set via project-level CI/CD variables. The executor type is arbitrary; # here is a minimal runnable config using docker. concurrent = 1 check_interval = 0 [[runners]] name = "test-runner" url = "https://your-gitlab.example.com" token = "<runner-token>" executor = "docker" [runners.docker] image = "alpine:latest" ``` </details> ### Used GitLab Runner version ``` Running with gitlab-runner 18.11.3 (ad1797b3) ``` (git version inside the helper image: `git version 2.47.3`) > Note: reproduced on 18.11.3, but the root-cause code (`head -n1`) has never been modified — the current `main` branch ([`shells/bash.go:304`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/main/shells/bash.go?ref_type=heads#L304)) still contains it, so **all versions are affected**. ## Possible fixes The root-cause code is in `shells/bash.go:277` (the `IfGitVersionIsAtLeast` function): ```go b.Line(`minimum_ver="$(printf '%s\n%s' "$required_ver" "$current_ver" | sort -t '.' -k 1,1n -k 2,2n -k 3,3n | head -n1)"`) ``` The problem is that `head -n1` closes the pipe after reading the first line, and the upstream `sort` receives SIGPIPE when writing the second line. **Suggested fix**: replace `head -n1` with `sed -n '1p'`. `sed` reads all input (it does not close the pipe early), so `sort` never receives SIGPIPE and the exit code is 0. The change is a single line: ```go // before b.Line(`minimum_ver="$(printf '%s\n%s' "$required_ver" "$current_ver" | sort -t '.' -k 1,1n -k 2,2n -k 3,3n | head -n1)"`) // after b.Line(`minimum_ver="$(printf '%s\n%s' "$required_ver" "$current_ver" | sort -t '.' -k 1,1n -k 2,2n -k 3,3n | sed -n '1p')"`) ``` **Verification**: reproduced inside the same helper image used in production (digest 571952e6, git 2.47.3): - Original version (`head -n1`): 10000 iterations, 7565 failures (exit code normalized to 1), output empty on failure. - Fixed version (`sed -n '1p'`): 30 iterations, all succeeded. **Alternative approaches** (all viable, but `sed -n '1p'` is the smallest change with the clearest semantics): - Append `|| true` to the pipeline to tolerate SIGPIPE. - Locally disable/re-enable pipefail around the version check (`set +o pipefail` / `set -o pipefail`).