PowerShell CI jobs on Kubernetes succeed despite failure in job script

Summary

After upgrading our GitLab Runner instances from 16.2.1 to 16.4.0, we found that PowerShell CI jobs whose scripts had an error exit code were shown as having succeeded (and allowing further pipeline steps to run). During our rollback procedure, we tested with several recent versions of GitLab Runner. We identified:

  • 16.2.1: works as expected
  • 16.2.2: failures are treated as successes
  • 16.3.0: works as expected
  • 16.3.1: failures are treated as successes
  • 16.3.2: failures are treated as successes
  • 16.4.0: failures are treated as successes
  • 16.4.1: failures are treated as successes

Steps to reproduce

Using a GitLab Runner configured for Kubernetes, run a pipeline with the below jobs.

.gitlab-ci.yml 
stages:
  - build
​
variables:
  POWERSHELL_IMAGE: mcr.microsoft.com/powershell:7.3-debian-bullseye-slim
​
default:
  interruptible: true
  image: $POWERSHELL_IMAGE
  tags:
    - eks-experimental-pwsh
​
should-succeed:
  stage: build
  script:
    - Write-Host "This should succeed."
  after_script:
    - Write-Host "After script."
​
should-fail:
  stage: build
  script:
    - Write-Host "This should fail."
    - throw "This should fail."
  after_script:
    - Write-Host "After script."

Actual behavior

The should-succeed job will succeed on all versions, and the should-fail will also succeed on versions listed above as "failures are treated as successes."

Expected behavior

The should-succeed job will succeed, and the should-fail job will fail.

Relevant logs and/or screenshots

The only difference in logs is the final line, indicating the success or failure of the job.

job log

16.2.2, 16.3.1, 16.3.2, 16.4.0, 16.4.1:

Job succeeded

16.2.1, 16.3.0:

ERROR: Job failed: command terminated with exit code 1

Environment description

Custom Installation of GitLab Runner, using Kubernetes executor and PowerShell shell, registered on GitLab.com

config.toml contents

(reduced to relevant information)

    [[runners]]
      executor = "kubernetes"
      shell = "pwsh"
      [runners.kubernetes]
        privileged = true
        helper_image_flavor = "ubuntu"

Used GitLab Runner version

Tested versions listed:

Running with gitlab-runner 16.2.1 (674e0e29)
Running with gitlab-runner 16.2.2 (adecbc65)
Running with gitlab-runner 16.3.0 (8ec04662)
Running with gitlab-runner 16.3.1 (d240d5bb)
Running with gitlab-runner 16.3.2 (77ba32bf)
Running with gitlab-runner 16.4.0 (6e766faf)
Running with gitlab-runner 16.4.1 (d89a789a)

Possible fixes

We took a look at the changes that were common to 16.2.2 and 16.3.1, and we believe this issue appears to be related to the changes introduced in !4315 (merged).

Edited by Adam Maras