gitlab runner on windows reports success when command not found

Summary

I have a gitlab CI job that runs on Windows build servers. With the current gitlab runner on Windows, jobs are failing, but being reported as successful:

Running with gitlab-runner 12.1.0 (de7731dd)
  on falconwinbld z4yiepra
Using Shell executor...

[...]

$ build_windows_ost_64.bat publish
The term 'build_windows_ost_64.bat' is not recognized as the name of a cmdlet, 
function, script file, or operable program. Check the spelling of the name, or 
if a path was included, verify that the path is correct and try again.
At C:\Users\GITLAB~1\AppData\Local\Temp\build_script858764150\script.ps1:147 ch
ar:25
+ build_windows_ost_64.bat <<<<  publish
    + CategoryInfo          : ObjectNotFound: (build_windows_ost_64.bat:String 
   ) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException
 
Job succeeded

Steps to reproduce

  1. Install a gitlab runner on Windows.
  2. Configure a CI job that has a bogus script.
  3. Push a commit and watch the CI job report success.

Here is the relevant .gitlab-ci.yml section:

deploy:windows:
    stage: deploy
    tags:
        - windows
    script:
        - build_windows_ost_64.bat publish
    only:
        - master

What is the current bug behavior?

The CI job reports success.

What is the expected correct behavior?

The CI job should report failure.

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:		Ubuntu 16.04
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.6.3p62
Gem Version:	2.7.9
Bundler Version:1.17.3
Rake Version:	12.3.2
Redis Version:	3.2.12
Git Version:	2.21.0
Sidekiq Version:5.2.7
Go Version:	unknown


GitLab information
Version:	12.1.2-ee
Revision:	d7aa668babd
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	10.7
URL:		http://gitlab.eng.local
HTTP Clone URL:	http://gitlab.eng.local/some-group/some-project.git
SSH Clone URL:	git@gitlab.eng.local:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers:


GitLab Shell
Version:	9.3.0
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...


Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 9.3.0 ? ... OK (9.3.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
26/1 ... yes
2/2 ... yes
2/3 ... yes
5/4 ... yes
5/5 ... yes
2/6 ... yes
2/7 ... yes
4/8 ... yes
3/9 ... yes
3/10 ... yes
5/11 ... yes
3/12 ... yes
5/13 ... yes
3/14 ... yes
3/15 ... yes
3/17 ... yes
4/18 ... yes
3/19 ... yes
2/20 ... yes
20/21 ... yes
20/22 ... yes
20/23 ... yes
20/24 ... yes
20/25 ... yes
20/26 ... yes
20/27 ... yes
20/28 ... yes
16/29 ... yes
20/30 ... yes
13/31 ... yes
13/32 ... yes
23/33 ... yes
25/34 ... yes
2/35 ... yes
15/36 ... yes
11/37 ... yes
4/38 ... yes
20/39 ... yes
11/40 ... yes
25/41 ... yes
6/42 ... yes
4/45 ... yes
6/47 ... yes
2/48 ... yes
2/49 ... yes
23/50 ... yes
29/51 ... yes
4/52 ... yes
2/53 ... yes
2/54 ... yes
25/55 ... yes
3/57 ... yes
11/59 ... yes
5/60 ... yes
11/63 ... yes
3/64 ... yes
3/65 ... yes
11/66 ... yes
3/67 ... yes
3/68 ... yes
3/69 ... yes
2/70 ... yes
3/71 ... yes
2/72 ... yes
33/73 ... yes
13/74 ... yes
26/75 ... yes
5/76 ... yes
4/77 ... yes
31/78 ... yes
3/80 ... yes
26/81 ... yes
26/82 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.3)
Git version >= 2.21.0 ? ... yes (2.21.0)
Git user has default SSH configuration? ... yes
Active users: ... 29
Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished