## Problem Today users are confused by inconsistent behaviour and mismatched expectations per UX studies some people expected "pass" means no vulnerabilities for a job. ### Goal Create consistency between all Secure and Protect scanners (analyzers), and set customer expectations. ### Proposal Change all analyzers to give 0 if they have no technical issues, even if there are findings, and enhance documentation around what a success means in user documentation. We can also, separate to the MVC, iterate on ideas to enhance the pipeline information with the pipeline team and look into what changes we are already planning within the MR to keep improving. ### Whats Next in time for 15.0 [Provide more information from secure analyzers](https://gitlab.com/groups/gitlab-org/-/epics/5727) ### Original Proposal - rejected Change all analyzers to exit with code 6 when findings are present and be consistent across scanners - which would allow green ✅ to mean no findings However, due to the currently available limited options for communicating pipeline job status as well as the current restrictions of the analyzers (no post processing to separate new findings vs existing) the decision was made to delay this work to 15.0 and work toward adding post-processing to avoid alert fatigue in users - at the expense of confusion which we hope to help with increased documentation. Today, analyzers are unaware of any finding that already exists in the `default` branch. If we switch to outputting exit code 6 on any finding in a feature branch, this means any vulnerability that already exists in the `default` branch—even if has been Dismissed—will cause the job to fail. Over time, this can lead to scanner jobs always failing for many projects, which is not a desirable outcome. That is why a post-processing step that can make these source/target branch comparisons and change the exit code accordingly is a necessary first step. <details><summary>Click to expand...</summary> ### Release notes Jibberish to trigger auto creation ## Problem Our AST scanners behaviour is not consistent, and also users are unclear what a failure and a success are. ## Proposal ### Change (in some cases) the meaning of success All AST scanners change to match the following meaning of success - A successful job means the analyzer(s) ran AND found nothing - A failed job means either - the job ran but one or more analyzer found something - one or more of the analyzers failed in an expected or unexpected way ### Implement consistent Exit Codes across AST as applicable add / update [Current secure exit codes](https://docs.gitlab.com/ee/development/integrations/secure.html#exit-code) Some specific codes will have a specific shared meaning across all AST for the purpose of helping customers more easily handle making changes to all AST items * 0 = Job succeeded and there were no findings [0 is currently what we say success is "Following the POSIX exit code standard, the scanner exits with 0 for success and any number from 1 to 255 for anything else. Success also includes the case when vulnerabilities are found."] * Job did not complete because it could not find target to scan * Job succeeded and there were findings * Job succeeded but was unable to produce an artifact * Job succeeded and but produced an unexpectedly empty artifact This is in no way a complete list and there will be other cases - "Job did not complete ..." in those other cases each team can and should implement as many unique codes as each team needs, when needed, but be specific and exit with error when you fail. ### Change the templates There is now a way to allow_failures for jobs with specified exit codes. [documentation](https://docs.gitlab.com/ee/ci/yaml/#allow_failureexit_codes) Each default template should change to allow the exit code `Job succeeded and there were findings` because [our own research shows this is needed](https://gitlab.com/gitlab-org/gitlab/-/issues/235772) The other failures should NOT be allow_failure by default ### Update the user documentation https://gitlab.com/gitlab-org/gitlab/-/issues/301123 ### Announcement blog post https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/73975 ### announcement release post https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/73972 ## Status ### SAST & Secrets - [ ] Success means no failure (all analyzers) - [ ] Add exit codes (at least `Job succeeded and there were findings`) - [ ] Change template & ADO to default to allow failure for 'Job succeeded and there were findings` - [ ] Link your users doc to new section in application_security section - [ ] Review blog post - [ ] review deprecation notice ### DAST - [ ] Success means no failure (all analyzers) - [ ] Add exit codes (at least `Job succeeded and there were findings`) - [ ] Change template & ADO to default to allow failure for 'Job succeeded and there were findings` - [ ] Link your users doc to new section in application_security section - [ ] Review blog post - [ ] review deprecation notice ### Protect - Container Scanning - [ ] Success means no failure (all analyzers) - [ ] Add exit codes (at least `Job succeeded and there were findings`) - [ ] Change template & ADO to default to allow failure for 'Job succeeded and there were findings` - [ ] Link your users doc to new section in application_security section - [ ] Review blog post - [ ] review deprecation notice ### Fuzzing - [ ] Success means no failure (all analyzers) - [ ] Add exit codes (at least `Job succeeded and there were findings`) - [ ] Change template & ADO to default to allow failure for 'Job succeeded and there were findings` - [ ] Link your users doc to new section in application_security section - [ ] Review blog post - [ ] review deprecation notice ### Composition Analysis - Dependency Scanning & License Compliance - [ ] Success means no failure (all analyzers) - [ ] Add exit codes (at least `Job succeeded and there were findings`) - [ ] Change template & ADO to default to allow failure for 'Job succeeded and there were findings` - [ ] Link your users doc to new section in application_security section - [ ] Review blog post - [ ] review deprecation notice </details>