AI Heuristic Evaluation Recommendation - Vulnerability Resolution
FigJam
Heuristic Evaluation:Resulting Recommendations
Discoverability | How do you find out this feature exists? If someone enables it for you, how do you find out you have it? |
- There is nothing on the report view that communicates the existance of this feature. |
Major |
Medium |
.com |
- Consider adding visual cues on table rows that communicates the existing of AI features. |
Learnability | After it is enabled, how do you access it the first time and learn how to use it? How easy is it to find it? Does the location match the task workflow? |
- Because there's no discoverability on the report view there's a potential misalignment with the user's workflow and therefore lowering their productivity potential. |
Major |
Medium |
.com |
- Consider allowing the user to click to resolve a vuln directly from the table row. |
Documentation and help | Are the docs findable, accurate and helpful? |
- Docs are innacurate |
Major |
Small |
.com |
- Recommend fixing Step 3 to more clearly explain that there's more to selecting SAST tools than simply selecting "SAST" You actaully have to click on each scanner you want to filter from SAST on. |
Recognize, diagnose and recover from errors | If I need help during set-up or first use, can I get it? |
- If unable to locate the source code for the selected vuln there's no info about what this means, how to fix it, or how to learn more about what's causing it in the alert. |
Major |
Small |
.com |
- We should provide a better string in the alert that provides more information. If possible explain how they can fix it. At the least we should link to docs to explain things. |
Communicating intent and context setting | How easy is it to initiate an interaction with the AI feature? How do you know what you can ask and how to ask it? |
- The primary button to access this feature's label isn't referencing GitLab Duo |
Low |
Small |
.com |
- The CTA label on the primary button should be updated to convert "AI" to "GitLab Duo" |
Reframing and tuning results | If I don’t get the best answer, how easy is it to improve it? Do I have to re-generate all output or can I update parts of it? |
- On the MR view to review the suggeseted resolution, there's no way for the user to attempt to refine the recommendation or to discuss it further. |
Major |
Large |
.com |
- We should allow Duo Chat functionality so the use can discuss the recommendation further. |
User control and freedom | When satisfied with the AI interaction, what’s the next step? If applicable, can you save, edit, insert, copy or otherwise use the output in a way that makes sense? |
- After the recommendation has been merged I'm unsure what happens to the vuln originally selected from the vuln report (I was unable eto test this out in the test project). |
Major |
Medium |
.com |
- We should clearly communicate that this vulln is in the process of being remediated and once it has been remediadted, mark it as resolved, and clear it from the report. |