SAST Config UI: Disabled analyzer warning
Problem to solve
An internal usability study surfaced a problem where the user thought he only had to enable the SAST analyzer associated with the language in the project. It was not clear that the analyzers only run if detected in the MR, even though the description to the SAST Analyzers area states this clearly. We want to encourage users to keep all of their analyzers on if possible so they get the most out of our SAST offering and don't miss any potential vulnerability findings.
Quotes
The following is the feedback we received on this topic from GitLab team members, as part of the SAST CMS study.
P2, Security Engineer:
I would tweak variables from the configuration UI... I still have to do a lot of thinking of which analyzers are relevant, etc...
&
P6, Software Developer:
I may want to disable an analyzer if I know I'll never use it, or that they might consume CI resources. My understanding is it needs to download a Docker package to detect the language.
Solution Validation (for internal usability testing)
Scenario: If you knew that there's no C/C++ in the project you're enabling SAST for, would you take any action from this SAST Configuration page?
Assumption: Most users would want to uncheck (disable) Flawfinder.
Ideal results: After user unchecks (disables) Flawfinder, they will see and read the alert above the Create a Merge Request button, and then decide to re-enable it.
Final proposal
Given the results of the internal usability test, let's proceed with the following alert if a user unchecks 1 or more of the analyzers in the SAST Configuration UI:
/cc @tmccaslin
