Proposal: let security products jobs succeed with warning on vulnerabilities found but still upload artifacts (when: on_failure)
Problem to solve
Secure products' jobs have to be "green" to upload artifacts but this gives a misleading view of a "green" pipeline to one who observes the pipeline widget. If security checks have failed on a pipeline, I as a user wish to see at least a warning.
- Adjust the security products' Docker images to exit with a non-zero code when vulnerabilities are found
- The zero exit code should be returned only if no vulnerabilities found and there was no fatal error during the scanning tool execution
- All security products' job will have
artifacts:when:on_failureset in respective vendored templates, Auto DevOps template, manual job definitions in the docs etc.
- This should be applied to GitLab.com as well (dogfooding)
What does success look like, and how can we measure that?
A user can instantly check that the security scanning tools have detected no vulnerabilities for his project by the green status of security products' jobs in the pipeline.
What is the type of buyer?
GitLab Ultimate users
Links / references
There is a consensus on that we should block MRs and not pipelines since GitLab Security Products' unique approach is to report newly introduced vulnerabilities and not just execute the security check over the codebase on each pipeline run.
Also, the job run and security report parsing and merging its data with dismissals information are 2 actions that belong in 2 different time frames. And the actual "no vulnerabilities" or "new vulnerabilities" found decision is made after the entire pipeline is complete. Thus invalidating the exit status of the security job as the source of the truth.
As a summary: the Security Gates seem really better alternative in the context of how GitLab Security Products operate and the fact that initial intent of this proposal was to just be more obtrusive for a user if vulnerabilities are found.