Update all fuzz jobs to fail when a vulnerability is reported
Problem to solve
Users find it confusing when a fuzz job finds vulnerabilities but is marked as succeeded in the pipeline view. We learned this behavior confuses people after testing with real users.
In this image, the job succeeded even though there were security vulnerabilities reported.
Proposal
Update all fuzz jobs (coverage-guided and API) templates to fail if a vulnerability is found. Add allow_failure:true
(or similar) to the template so that the job can fail but the overall pipeline won't fail.
Note: Some fuzz languages already use this convention.
Note: Similar issues as this will be created for other Secure scanners. The scope of this issue is just fuzz testing.
Edited by Sam Kerr