DESIGN: Make Secure analyzers exit with non-zero code when vulnerabilities are found
Release notes
Problem to solve
As a user inspecting the pipelines of my GitLab projects, I want to easily see that vulnerabilities have been found by a scanning job. Also, I want to be able to make the pipeline fail when vulnerabilities are found. Finally, I expect this behavior to consistency across all GitLab security scanners.
Intended users
User experience goal
Users inspecting their project pipelines should see the scanning jobs as failing when vulnerabilities are found.
Proposal
Make Secure analyzers exit with a specific exit code when vulnerabilities are found, and change the CI templates to allow the scanning jobs to fail with that particular exit code.
Exit code when vulnerabilities are detected: 6
The exit codes returned by the security scanners are versioned, just like the security reports. Scanner implementing exit codes v1 returns the expected exit code when vulnerabilities are detected.
User can override the definition of the scanning job and reset allow_failure:exit_codes
so that it's not allowed to fail when vulnerabilities are found.
Environment variable
A new CI variable named SECURE_ANALYZERS_EXIT_CODES_VERSION
controls the exit codes to be returned by the scanner.
- if
0
or not set, the scanner returns the legacy exit codes - if
1
, it returns6
when vulnerabilities/crashes are found
The latter corresponds to the version 1 of the Security scanner exit codes.
The environment variable that defines the exit code to be used when vulnerabilities are found is Scanners also provide an alias for this environment variable, with a prefix that corresponds to the product category, like Previous proposal: VULNERABILITIES_DETECTED_EXIT_CODE (integer)
VULNERABILITIES_DETECTED_EXIT_CODE
. It defaults to 0
, which corresponds to the existing behavior.SAST_VULNERABILITIES_DETECTED_EXIT_CODE
for SAST. This way the behavior can be changed only for one type of scanner.
Dismissed vulnerabilities
By design, a scanning job is unaware of vulnerabilities that have been dismissed in the UI, so it might fail because of false positives.
Allowlist
When the scanner supports some kind of vulnerability filter or allowlist, vulnerabilities that have been excluded are NOT considered to determine the exit code. The exit code should thus be 0
(success) if all vulnerabilities found by the scanner have been excluded.
Migration path
This new behavior is enabled only in new versions of GitLab but some scanners like SAST are deployed to older versions of GitLab when updated. (This will change as part of &4060 (closed).) To achieve backward compatibility, the new behavior is enabled when a newly introduced environment variable is set, and it's set in the CI templates that ship with GitLab. As a result older versions of GitLab are not impacted by the change.
For security scanners like SAST that consist of multiple analyzers and multiple scanning jobs, the migration is a 2-step process:
- update the analyzers so that they support the new behavior
- change the base job in CI template, in order to enable the new behavior, and allow the new exit codes
Doing so ensures that all scanning jobs of the same product category behave the same way.
When publishing a new major version of the analyzer, we might decide to change the default value of the environment variable that controls the behavior, and make analyzers fail by default when vulnerabilities are found.
In the future, the exit codes version will be bumped when introducing new exit codes or to redefining the existing ones. If supported by the analyzer, CI variable SECURE_ANALYZERS_EXIT_CODES_VERSION
provides an easy migration path when changing the exit codes.
Rails backend
The Rails backend is changed so that there's no warning in the Vulnerability Report
page when scanning jobs fail and are allowed to fail. It currently shows: N failed security jobs
along with the link to the pipeline.
The Rails backend is changed so that the Dependency List
shows dependencies even if the Dependency Scanning jobs fail and are allowed to fail. It currently shows The dependency_scanning job has failed and cannot generate the list. Please ensure the job is running properly and run the pipeline again.
Analyzers implemented in Go
For analyzers implemented in Go and using urfave/cli
, the Go function that implements the run
command is changed so that it returns an error when the report contains vulnerabilities. The error implements the ExitCoder
interface to communicate the exit code, and the error isn't logged, as explained in #233452 (closed).
The run
command is implemented in the command package if the Go project depends on it, or else in the analyzer itself. command was recently extracted from the common repo. See gitlab-org/security-products/analyzers/common!145 (diffs)
Security report
The security report itself isn't changed; .scan.status
is success
whenever the scan successfully completes, even when vulnerabilities are found.
Implementation plan
- Update the CI templates used for QA, capture the exit code, and test it
- Update the
Run
function in the command package so that it exits with a specific exit code when theReport
contains vulnerabilities; the behavior is controller by a new environment variable, and it's disabled by default to ensure backward compatibility - SAST
- Update CI template used for QA, test exit code
- Update analyzers; upgrade
common/command
or port the change- security-code-scan
- flawfinder
- brakeman
- eslint
- spotbugs
- gosec
- bandit
- phpcs-security-audit
- sobelow
- pmd-apex
- kubesec
-
nodejs-scan (doesn't depend on
run
)
- Update CI template to enable new exit codes, and allow them
- Update user docs
- Dependency Scanning
- Update the Rails backend so that the Dependency List is rendered even when Dependency Scanning jobs fail, as long as the generated reports are valid
- Update CI template used for QA, test exit code
- Update analyzers; upgrade
common/command
or port the change - Update CI template to enable new exit codes, and allow them
- Update user docs
- Secret Detection
- Update CI template used for QA, test exit code
- Update analyzers; upgrade
common/command
or port the change - Update CI template to enable new exit codes, and allow them
- Update user docs
- Container Scanning
- Update CI template used for QA, test exit code
- Update analyzers; upgrade
common/command
or port the change - Update CI template to enable new exit codes, and allow them
- Update user docs
- Update Security Scanner Integration guide
- Update Application Security docs
Permissions and Security
No change
Documentation
The default behavior (failing job when vulnerabilities are found) and its customization (to make the pipeline fail) should be documented in Application Security docs, and possibly in the user docs of the product categories affecting by the change (SAST, Dependency Scanning, etc.).
To be documented as a guideline in Security Scanner Integration guide.
Availability & Testing
Job integration tests (AKA "Secure QA") need to check the exit code of the scanning job. This should cover both the case when vulnerabilities are found and when the project is sane.
Currently the exit code isn't checked, and there are no tests for when the project has no vulnerabilities.
Available Tier
What does success look like, and how can we measure that?
Users reviewing a pipeline can easily spot scanning jobs where vulnerabilities are detected.
What is the type of buyer?
Is this a cross-stage feature?
Yes, this is a section-wide feature.