馃帹 Design: Update secure job status with corresponding exit code with correct icons
Background
This is created by old issue because the original one has too many diverse discussions. From the latest comment thread, we came out with the action point so we created this new one to focus the discussion.
Problem
Users can set "allow_failure" in security jobs, which means Allow job to fail. A failed job does not cause the pipeline to fail
Current behaviours
Research result:
- Green checkmarks doesn't communicate there are vulns found at all
- Yellow! marks communicate the best there are vulns found
- Red marks communicate the best there are vulns found
Key Problems:
- Default
allow_failure
not set, which will trigger either a Red X or a Green checkmarks, none of them is ideal - Our tooltip message is not clear enough to tell why job fails
- Our tooltip message is not consistent for the same status icon and shows at different pages
Additional problem:
See problem 2 in the comment: #300415 (comment 497223719)
Proposed solutions
- "allow_failure" by default
- Update doc, security
- We need to inform current users about the change
- update icon and tooltip
- tooltip should come from backend end, all icons with the same job same status, should have same tooltip
Update in g-doc: https://docs.google.com/spreadsheets/d/1cFKGRMxA59aWPiao_bJYgVgR4fWmaqO67_nDWGeiolk/edit?usp=sharing
Additional note:
@stkerr Please correct me, if I am wrong. This issue will ONLY be updated with coverage/API fuzz testing. We hope all scanners could reach an agreement for consistency, but we also understand that different scanners will have different concerns.