Windows PowerShell failed still marks job successful

Summary

A Powershell command failed, job stopped (while there are some other script lines) but it succeeded. It mustn't.

Steps to reproduce

Remove an item with Powershell : Remove-Item ***

Actual behavior

Job succeeds

Expected behavior

Job fail

Relevant logs and/or screenshots

$ cd build
$ Remove-Item "CMakeCache.txt"
Remove-Item : Cannot find path 'C:\builds\ee766567\0\toto\common\build\CMakeCache.txt' because 
it does not exist.
At C:\Users\toto\AppData\Local\Temp\build_script825588960\script.ps1:141 char:1
+ Remove-Item "CMakeCache.txt"
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\builds\ee766...\CMakeCache.txt:String) [Remove-I 
   tem], ItemNotFoundException
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.RemoveItemCommand

Job succeeded

Environment description

Windows 10

Used GitLab Runner version

Running with gitlab-runner 10.6.0 (a3543a27) on Windows 10 - Desktop PC Toto ee766567 Using Shell executor...

Shell is Powershell

Root Cause

PowerShell has different mechanisms on how to handle errors using the ErrorActionPreference. The default value of this inside of PowerShell is set to Continue so if PowerShell script fails it will continue and not report it as an error. If ErrorActionPreference is set to Stop it will stop as expected, as shown here.

GitLab Runner should set this value to Stop so users get the expected behavior, but we need to be careful this can be considered a breaking change because it will break existing passing pipelines.

Workaround

Set ErrorActionPreference to Stop inside of the .gitlab-ci.yml of variables

job:
  stage: test
  variables: 
    ErrorActionPreference: stop 
  script:
  - NONEXISTANTCMDLET