Proposal: let security products jobs succeed with warning on vulnerabilities found but still upload artifacts (when: on_failure)

Problem to solve

Secure products' jobs have to be "green" to upload artifacts but this gives a misleading view of a "green" pipeline to one who observes the pipeline widget. If security checks have failed on a pipeline, I as a user wish to see at least a warning.

Intended users

Persona: Security Analyst

Further details

Proposal

• Adjust the security products' Docker images to exit with a non-zero code when vulnerabilities are found
• The zero exit code should be returned only if no vulnerabilities found and there was no fatal error during the scanning tool execution
• All security products' job will have artifacts:when:on_failure set in respective vendored templates, Auto DevOps template, manual job definitions in the docs etc.
• This should be applied to GitLab.com as well (dogfooding)

Documentation

TODO

What does success look like, and how can we measure that?

A user can instantly check that the security scanning tools have detected no vulnerabilities for his project by the green status of security products' jobs in the pipeline.

What is the type of buyer?

GitLab Ultimate users

Links / references

Current status

There is a consensus on that we should block MRs and not pipelines since GitLab Security Products' unique approach is to report newly introduced vulnerabilities and not just execute the security check over the codebase on each pipeline run.

Also, the job run and security report parsing and merging its data with dismissals information are 2 actions that belong in 2 different time frames. And the actual "no vulnerabilities" or "new vulnerabilities" found decision is made after the entire pipeline is complete. Thus invalidating the exit status of the security job as the source of the truth.

As a summary: the Security Gates seem really better alternative in the context of how GitLab Security Products operate and the fact that initial intent of this proposal was to just be more obtrusive for a user if vulnerabilities are found.