Change License Finder to error when findings found and add "Job succeeded and there were findings" exit code
Problem to solve
Users want to have a consistent experience within GitLab as it is one product for them.
In user research we found that users are sometimes confused that a "passed" job contains findings
We have heard (in the form of user reported bugs) both of the above situations where people believe the product is broken because of an inconsistency, or the "passing" of some jobs with findings and failings of others with findings.
As a result we wish to make sure all analyzers exit with a specific error code when "Job succeeded and there were findings"
Question
Can we do it under a feature flag or something that we flip in 14.0 or version it so it doesn't break anyone until they change on purpose?
We'll want the documentation and template changes etc etc all at the same time at least
OTHER ISSUES (fast follow or in parallel where possible?)
- We also want to Change default templates to allow_failure with "Job succeeded and there were findings" so that we don't break anyones pipelines!
- Change AutoDevOps to allow_failure with "Job succeeded and there were findings"
- Update docs for SCA to link to application_security section on success/failure
- blog
- release post as well
Intended users
User experience goal
users will now see a warning (if allow error) or error (if not allow error) by default if findings are found by an analyzer, better matching their expectations.
Proposal
Next available number is 6? or we can grab 10 and reserve 1X for related error codes? https://docs.gitlab.com/ee/development/integrations/secure.html#exit-code
Further details
Permissions and Security
no changes
Documentation
update error code docs
Update changelog
Availability & Testing
test all the things!
Is this a cross-stage feature?
Keep all of secure and protect in the loop!
Links / references
Implementation Plan
Decide if the following issues are needed, or feel free to close them out if not
- Change retire.js to error when findings found and add "Job succeeded and there were findings" exit code
- Change bundler-audit to error when findings found and add "Job succeeded and there were findings" exit code
- Change gemnasium-python to error when findings found and add "Job succeeded and there were findings" exit code
- Change gemnasium-maven to error when findings found and add "Job succeeded and there were findings" exit code