Exit code improperly reported using the Bash script with `set -e`

Summary

Bash's set -e is quite inconsistent, and in our current usage in the Bash script generation of:

	_, _ = io.WriteString(w, "set -eo pipefail\n")
	_, _ = io.WriteString(w, "set +o noclobber\n")
	_, _ = io.WriteString(w, ": | eval "+helpers.ShellEscape(b.String())+"\n")

There is the possibility that a binary returning an exit code will wrongfully return the exit code 1.

For example, this simple job - for cat/repro-docker-exitcode@20f28073:

image: python:latest

test1:
  stage: test
  script:
    - python -c "import sys; sys.exit(42)" # same would happen for bash -c "exit 42" or others

Testing some more locally the bash behavior, it looks like the combination of pipeline + eval returns the wrong exit code:

[catalin@thetis bash]$ bash -c 'set -e; : | bash -c "exit 42"'; echo $?
42
[catalin@thetis bash]$ bash -c 'set -e; : | eval "exit 42"'; echo $?
42
[catalin@thetis bash]$ bash -c 'set -e; eval "bash -c \"exit 42\""'; echo $?
42
[catalin@thetis bash]$ bash -c 'set -e; : | eval "bash -c \"exit 42\""'; echo $?
1
[catalin@thetis bash]$ bash -c 'set -e; : | eval "eval \"bash -c \\\"exit 42\\\"\""'; echo $?
1

We can also quickly recompile bash and notice that bash sees another SimpleCommand forked *before* the child process (i.e. more or less, eval itself) returning exit code 1: (expand for details + strace of the behavior)

modified   execute_cmd.c
@@ -885,6 +885,11 @@ execute_command_internal (command, asynchronous, pipe_in, pipe_out,
 	       subshells forked to execute builtin commands (e.g., in
 	       pipelines) to be waited for twice. */
 	      exec_result = wait_for (last_made_pid);
+      WORD_LIST *w;
+
+      for (w = command->value.Simple->words; w; w = w->next)
+        fprintf(stderr, "%s%s", w->word->word, w->next ? " " : "");
+      fprintf(stderr, "\nEXIT: %d\n", exec_result);
 	  }
       }

[catalin@thetis bash]$ ./bash -c 'set -e; : | bash -c "exit 42"'
bash -c "exit 42"
EXIT: 42
[catalin@thetis bash]$ ./bash -c 'set -e; eval "bash -c \"exit 42\""'
bash -c "exit 42"
EXIT: 42
[catalin@thetis bash]$ ./bash -c 'set -e; : | eval "exit 42"'
eval "exit 42"
EXIT: 42
[catalin@thetis bash]$ ./bash -c 'set -e; : | eval "bash -c \"exit 42\""'
bash -c "exit 42"
EXIT: 42
eval "bash -c \"exit 42\""
EXIT: 1

strace -fTttyyy -s1024 -o /tmp/strace2 bash -c 'set -e; : | eval "bash -c \"exit 42\""'
# =>
1135  21:28:48.243654 exit_group(42)    = ?
1135  21:28:48.244046 +++ exited with 42 +++
1134  21:28:48.244096 <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 42}], 0, NULL) = 1135 <0.050219>
1134  21:28:48.244263 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1135, si_uid=0, si_status=42, si_utime=0, si_stime=0} ---
1134  21:28:48.244299 wait4(-1, 0x7fff2c2faa50, WNOHANG, NULL) = -1 ECHILD (No child processes) <0.000014>
1134  21:28:48.244472 exit_group(1)     = ?
1134  21:28:48.244663 +++ exited with 1 +++
1132  21:28:48.244691 <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, NULL) = 1134 <0.053271>
1132  21:28:48.245213 exit_group(1)     = ?
1132  21:28:48.245397 +++ exited with 1 +++
# where 
╰─>$ strace-parser strace2 exec

Programs Executed

      pid               program                args
  -------              ---------               ------
     1132              /bin/bash               ["-c" "set -e; : | eval \"bash -c \\\"exit 42\\\"\""] 0x7ffcf69c1000
     1135              /bin/bash               ["-c" "exit 42"] 0x5571d503d2a0

Steps to reproduce

.gitlab-ci.yml

image: python:latest

test1:
  stage: test
  script:
    - python -c "import sys; sys.exit(42)"

Actual behavior

We report the exit code back to GitLab Rails as 1.

Expected behavior

The actual exit code should be reported back.

Relevant logs and/or screenshots

job log

$ python -c "import sys; sys.exit(42)"
Cleaning up file based variables
00:00
ERROR: Job failed: exit code 1

Possible fixes

One possibility, it seems - is to wrap the eval in its own subshell, like:

[catalin@thetis bash]$ ./bash -c 'set -e; : | (eval "bash -c \"exit 42\"")'
bash -c "exit 42"
EXIT: 42