Exit code of Brakeman is ignored

Summary

The SAST Brakeman analyzer ignores the exit code (exit status) of the brakeman command. When brakeman fails, it complains about some missing /tmp/brakeman.json file, which is misleading.

Steps to reproduce

Run the SAST Brakeman analyzer on a project that's not compatible with Brakeman even though it has a Gemfile.lock and depends on "rails" - it's necessary to meet the two conditions otherwise the compatibility check performed by the wrapper prior to running brakeman won't pass.

Example Project

https://gitlab.com/joshlambert/spree

See https://gitlab.com/joshlambert/spree/-/jobs/74456586

What is the current bug behavior?

It complains about missing brakeman.json file:

$ docker run -ti --rm --volume $PWD:/tmp/app --env CI_PROJECT_DIR=/tmp/app registry.gitlab.com/gitlab-org/security-products/analyzers/brakeman:11-0-stable 
Found project in /tmp/app
2018/06/13 06:41:23 open /tmp/brakeman.json: no such file or directory

What is the expected correct behavior?

It should say that the brakeman command has failed.

Possible fixes

Right now the SAST Brakeman ignores the exit code of brakeman because it's non-zero when Brakeman finds a vulnerability, which is not what we want. After further investigation, it looks like cli options make it possible to change this behavior:

brakeman --no-exit-on-warn --no-exit-on-error

Not critical but I suggest we fix this issue in the next release because it makes it harder to debug the Brakeman analyzer.