Relates to [#inc-12090-dev-runners-erroring-out-on-step-concrete](https://gitlab.enterprise.slack.com/archives/C0BHQLLEYJX) ## Summary Under `FF_CONCRETE`, the concrete step-runner's bash script generator wraps every user-script line with a `$?`-restore prologue (`(exit $_runner_exit_code)`). Because the whole script runs under `set -o errexit`, this restore becomes a standalone `(exit 1)` at the top of any `else`/`elif` branch — carrying the failed `if`-condition's exit status — which trips `errexit` and **aborts the job before the user command in that branch ever runs**. This is a concrete-only regression: the legacy shell path does not emit the restore line, so it is unaffected. It fires **regardless of `FF_ENABLE_BASH_EXIT_CODE_CHECK`**. Discovered while debugging a real failure: `dev.gitlab.org/gitlab/charts/components/images` job `#37766504` (docker+machine executor, `FF_CONCRETE:true`), which failed with: ``` ERROR: Job failed: step "concrete": exec: step script:exit code 1 ``` ## Root cause `functions/concrete/run/stages/internal/scriptwriter/scriptwriter.go` (`buildBashScript`, ~lines 60–106) emits, for **every** user-script physical line: ```sh _runner_exit_code=$? # :79 — capture previous command's $? echo "<green $ header>" # :88 / :101 — echoing the trace header resets $? to 0 (exit $_runner_exit_code) # :93 / :104 — "restore" $? for the user command <user line> # :94 / :105 ``` and runs the script under `set -o errexit` (`:114`). `(exit N)` with `N != 0` is itself a command with non-zero status, so under `errexit` it aborts the script. This becomes fatal when the prologue is injected as the first statements inside an `else`/`elif` branch: on entering `else`, `$?` holds the (non-zero) exit status of the failed `if` condition, so `_runner_exit_code=1` and the injected `(exit 1)` fires `errexit`. - `then` branches are safe (entered with `$?`=0). - `if` with a false condition and **no** `else` is safe (an `if` returns 0 overall). - The **`else` / `elif` branch is the trap**. A trailing `! cmd` on a logical line is similarly affected. ## Steps to reproduce Minimal reproduction of the generated pattern (exits 1; `END` and the else body never print): ```sh bash -c 'set -e; if [ "$X" = "true" ]; then :; else _r=$?; echo hdr; (exit $_r); echo NEVER; fi; echo END' ``` Any `FF_CONCRETE` job whose `script:` contains an `if [ … ]; then …; else <cmd>; fi` where the `else` branch is taken will fail. ## Evidence (from a `CI_DEBUG_TRACE` re-run) Generated script (prologue injected inside the branch): ``` if [ "$UBI_PIPELINE" = "true" ]; then _runner_exit_code=$? echo "$ else" (exit $_runner_exit_code) else _runner_exit_code=$? # <- $? = 1 (failed if-condition) echo "$ logger_version=$(get_logger_version)" (exit $_runner_exit_code) # <- (exit 1) trips set -e logger_version=$(get_logger_version) ... fi ``` xtrace at the failure point: ``` ++ '[' '' = true ']' # if-condition, false (exit 1) ++ _runner_exit_code=1 # captured at top of else branch ++ echo '$ logger_version=$(get_logger_version)' ++ exit 1 # (exit $_runner_exit_code) -> set -e ABORT section_end:...:step_script # logger_version=... NEVER RAN ``` ## Impact Broad. Any concrete job whose `script:` uses an `if/else` (or `if/elif`, or trailing `! cmd`) where the else/elif branch is entered will abort silently — the user command in that branch never runs, and the log shows only its echoed `$ …` header followed by the failure. It fires regardless of `FF_ENABLE_BASH_EXIT_CODE_CHECK`. ## Proposed fix File: `functions/concrete/run/stages/internal/scriptwriter/scriptwriter.go` (`buildBashScript`). The `(exit $_runner_exit_code)` status-restore is fundamentally unsafe under `set -o errexit`. 1. **Recommended (legacy parity):** drop the pre-command `(exit $_runner_exit_code)` restore (and the paired `_runner_exit_code=$?` capture where it only feeds that restore). User commands then observe `$?`=0 after the header echo — exactly the legacy behavior — eliminating the errexit trap. Keep the separate FF-gated `checkErr` capture that runs *after* the command. 2. If preserving `$?` for the first user command is genuinely required, do it without a standalone non-zero command (e.g. restore only in an `errexit`-exempt position, or inline it with the user command so `errexit` evaluates only the user command). Add regression coverage in `functions/concrete/run/stages/internal/scriptwriter/scriptwriter_test.go`: a script with `if [ x = y ]; then …; else <cmd>; fi` (else branch taken), asserting the script exits 0 and the else body runs. Cross-check equivalent legacy behavior in `shells/bash_test.go`. ## Notes - The charts CI script is **not** at fault; `UBI_PIPELINE` being unset is merely the trigger condition that exposes the runner bug. - Secondary observation: in the failing job, `upload_artifacts_on_failure` was skipped as *"not applicable for current job status"* — under concrete, failure is only recorded after the whole step returns, so on-failure artifact upload may not fire for script failures. Likely a separate, lower-priority follow-up.