semgrep-sast should fail with invalid ruleset config
Summary
Our semgrep-sast
job does not currently fail when provided with an invalid configuration file, but it should.
It's unclear if this is widespread or semgrep specific, we should investigate further.
Implementation plan
- DRY
ProcessPassthrough
handling for all analyzers - Correct https://gitlab.com/gitlab-org/security-products/post-analyzers/scripts to return proper exitcode
- Update analyzers to latest
scripts
version (gitlab-org/security-products/analyzers/gosec!131 (merged) and gitlab-org/security-products/analyzers/brakeman!88 (merged)) - Remove unmaintained
tracking-calculator
start.sh
(https://gitlab.com/gitlab-org/security-products/post-analyzers/tracking-calculator/-/merge_requests/34)
Steps to reproduce
- Set
.gitlab/sast-ruleset.toml
to a nonsense value - Run
semgrep-sast
job - Observe job still passes
Example run
❯ docker run --rm --volume "$PWD":/tmp/app --env GITLAB_FEATURES="sast_custom_rulesets,vulnerability_finding_signatures" --env CI_PROJECT_DIR="/tmp/app" semgrep:main
[INFO] [Semgrep] [2021-10-25T21:35:51Z] ▶ GitLab Semgrep analyzer v2.13.3
[INFO] [Semgrep] [2021-10-25T21:35:51Z] ▶ Detecting project
[INFO] [Semgrep] [2021-10-25T21:35:51Z] ▶ Found relevant files in project, analyzing entire repository
[INFO] [Semgrep] [2021-10-25T21:35:51Z] ▶ Running analyzer
[FATA] [Semgrep] [2021-10-25T21:35:51Z] ▶ stat /tmp/app/semgrep_rulesssss: no such file or directory
running post analyzer
[INFO] [2021-10-25T21:35:51Z] ▶ /tmp/app/gl-sast-report-post.json written
❯ echo $?
0
Example Project
https://gitlab.com/theoretick/spotbugs-semgrep/-/jobs/1648017107
What is the current bug behavior?
Analyzer jobs don't fail with invalid ruleset configuration
What is the expected correct behavior?
Analyzer jobs should fail with invalid ruleset configuration
Relevant logs and/or screenshots
See example project
Output of checks
This bug happens on GitLab.com
Edited by Lucas Charles