Exit Code of executed program is ignored

Summary

Im using a python script to lint a LUA application. The application basically lints the LUA code and exits with 0 if everything is fine. If not, it uses exit code 1 on any critical lua error and exit code 2 if a syntax error occures.

The problem, the exit code of the python call is not used / honored. If it fails, it always fails with exit code 1

Steps to reproduce

I created a simple repository to reproduce this issue: https://gitlab.com/SeaLife/gitlab-runner-bug

If i use exit 3 as statement, it works, but if i use sys.exit(3) in the python code, its not working. Using the Feature-Flag (or not using it) does not make any difference

.gitlab-ci.yml

stages:
    - build

variables:
    FF_ENABLE_BASH_EXIT_CODE_CHECK: "true"

build application exit code 1:
    stage: build
    image: python:3.7
    script:
        - python3 ex1.py
    allow_failure:
        exit_codes:
            - 1
            - 2

build application exit code 2:
    stage: build
    image: python:3.7
    script:
        - python3 ex2.py
    allow_failure:
        exit_codes:
            - 1
            - 2

build application exit code 3:
    stage: build
    image: python:3.7
    script:
        - python3 ex3.py
    allow_failure:
        exit_codes:
            - 1
            - 2


build application bash exit code 1:
    stage: build
    image: python:3.7
    script:
        - exit 1
    allow_failure:
        exit_codes:
            - 1
            - 2

build application bash exit code 2:
    stage: build
    image: python:3.7
    script:
        - exit 2
    allow_failure:
        exit_codes:
            - 1
            - 2

build application bash exit code 3:
    stage: build
    image: python:3.7
    script:
        - exit 3
    allow_failure:
        exit_codes:
            - 1
            - 2

Actual behavior

The Python3 Interpreter exits with ExitCode 2 but the Gitlab Runner reports "ERROR: Job failed: exit code 1"

Expected behavior

I expect to get the exit code reported by sys.exit(n)

Relevant logs and/or screenshots

job log

Running with gitlab-runner 14.8.0~beta.44.g57df0d52 (57df0d52)
  on blue-4.shared.runners-manager.gitlab.com/default J2nyww-s
  feature flags: FF_ENABLE_BASH_EXIT_CODE_CHECK:true
Preparing the "docker+machine" executor
00:29
Using Docker executor with image python:3.7 ...
Pulling docker image python:3.7 ...
Using docker image sha256:dbd6ecf8172a4b60c630ee42c9f351ec6496aff8b98dafd9a2539171ea07fd32 for python:3.7 with digest python@sha256:d577c8019e75b50e71e074449ef8fb6058bfdd45e12e6336972256b345310983 ...
Preparing environment
00:03
Running on runner-j2nyww-s-project-34205274-concurrent-0 via runner-j2nyww-s-shared-1646384444-7ee29df7...
Getting source from Git repository
00:02
$ eval "$CI_PRE_CLONE_SCRIPT"
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/SeaLife/gitlab-runner-bug/.git/
Created fresh repository.
Checking out d435c0b6 as main...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:01
Using docker image sha256:dbd6ecf8172a4b60c630ee42c9f351ec6496aff8b98dafd9a2539171ea07fd32 for python:3.7 with digest python@sha256:d577c8019e75b50e71e074449ef8fb6058bfdd45e12e6336972256b345310983 ...
$ python3 ex3.py
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

Environment description

Using a shared runner on GitLab.com. Using a custom runner does not make any difference.

Used GitLab Runner version

Running with gitlab-runner 14.8.0~beta.44.g57df0d52 (57df0d52)
  on blue-4.shared.runners-manager.gitlab.com/default J2nyww-s
  feature flags: FF_ENABLE_BASH_EXIT_CODE_CHECK:true

Possible fixes

🤷