馃帹 Design/Research: Consider failing secure jobs when vulnerability findings are found
Problem to solve
Because our Secure jobs pass even when vulnerabilities are found, users may reference the pipeline view, see a green check, and assume they are secure when they are in fact not.
We have received this feedback multiple times that the current behavior is not intuitive. We also have heard the exact opposite feedback that we should continue passing jobs and that users should be required to reference the pipeline security tab.
User experience goal
Behavior more closely aligned with user expectations - that vulnerability findings should equal failing pipelines
Proposal
Conduct user research to (in)validate what end-users' perceptions are.
After researching and if the findings are that users are confused with our current approach, change the default behavior of security scanners to fail when vulnerability findings are found but keeping the allow_failure: true.
- This use of
allow_failureallows us to draw additional visibility to the results of the security job while not blocking the pipeline, which is one of our UX goals in Secure.
Further Details
Specific scenarios to investigate:
- Scenario A - No findings found
- Scenario B - findings found
- Scenario C - Scanner failed due to exception
Permissions and Security
No change to permissions
Documentation
Availability & Testing
What does success look like, and how can we measure that?
What is the type of buyer?
Is this a cross-stage feature?
While this discussion is focused on devopssecure there is potential for it to involve other stages; i.e. Category:Code Quality within devopsverify