Solution Validation: Group-level security scan status
What's this issue all about? (Background and context)
In gitlab#262079 (closed) we arrived at a design solution for the problem of: how do I know that my security scans are enabled and running successfully across projects? We now want to perform problem and solution validation.
What are the overarching goals for the research?
- Validate the need for this widget
- Validate that the solution solves a problem
- Validate that building this will be worth our team's resources, especially engineering
- Identify areas of improvement in the proposed solution, if anything
- Help our internal team to understand why and how this feature will be used
What hypotheses and/or assumptions do you have?
- The information in this widget will be particularly helpful as a starting point to see where there might be issues in the implementation of security scanners across a group
- The design and interaction will be intuitive and a value add, worthy of our team's resources that it will require to build it out
What research questions are you trying to answer?
- Given scenarios, can the users complete them using the UX and UI of this particular design?
- Is this solving a problem for the user?
- If yes, is this the best way to solve the problem?
- Is the information intuitive? (Do they associate the icons with each job ran, are the meanings of the icons intuitive and/or do they think to hover for more information, does the tooltip provide the right information, etc)
- What filters would they want to apply?
- What would they do next? (Do the links and subsequent workflows make sense?)
- Is anything missing?
What persona, persona segment, or customer type experiences the problem most acutely?
What business decisions will be made based on this information?
Put resources towards building and maintaining the widget
What, if any, relevant prior research already exists?
Who will be leading the research?
Becka, via usertesting.com
What timescales do you have in mind for the research?
Minimum 5 participants by end of %13.11
Relevant links (script, prototype, notes, etc.)
-
Complete Research Plan -
Review with Tali, Andy, Matt, make any necessary updates -
Build in usertesting.com -
Send to qualified usertesting audience, any relevant customers, internal GitLab security engineers -
Summarize findings -
Create followup design issue(s) to address UX/UI concerns, if any
Result overview
| Participant # | Role | Went well | What didn't go well/ Suggestions for improvement | Overall score (1-7) |
|---|---|---|---|---|
| 1 | Security Professional/ Analyst | He understood the value of the widget and the meaning of the icons | Thought there could be a key explaining the meaning of each icon (as he was unsure what one of them meant (skipped)), but the prototype didn't have tooltips over each icon which explains that, so I think we can disregard this suggestion; Wanted to see the reason for failure in the "Failed job" tab (not sure how often we have this info there but this is tangential feedback for this study) | 6 |
| 2 | Full-stack Developer (in the screener but in her participant profile says Full Time Student) | Likes that she can see all the tests and which have failed and succeeded, likes the colors (e.g. red for fail), easy to enable SAST from Configuration page | Expected to go to "Project D" link and see why the project failed there (instead it goes to the Configuration page which does not have pipeline info), was not expecting to see log files when she clicked on the job failure so she didn't think that part was intuitive (not relevant for this study) | 4 |
| 3 | Software Engineer/ Developer | Filters were easy to use and intuitive, the workflow (linked pages) matched his expectations | Interpreted my scenario saying to look at projects where SAST "didn't run" as "where SAST failed", so tried to search for "failed" instead of "not found"; would prefer icons to be all vertically rather than 2 per line (but he kept switching between horizontally and vertically so I'm not sure which he really meant) | 6 |
| 4 | Software Engineer/ Developer | Would be useful for him to have a dashboard like this in his work, workflow feels intuitive | Says at the end that he wanted to see the vulnerability info itself (and suggestions for fixing it) on the failed jobs page (even though he didn't mention this when he was looking at the failed job tab of the pipeline); interpreted my scenario saying to look at projects where SAST "didn't run" as "where SAST was skipped", so tried to select "skipped" instead of "not found" in the filter dropdown, but when he saw that "skipped" wasn't enabled in the prototype he thought about it more and then chose "not found"; wonders what would happen if 5+ jobs ran per scanner- if there were 10 jobs he wonders what would happen (but he doesn't say what his preference would be on how to show that; he might be worried that if there's more than 4 they don't show up at all?) - he changes his score from a 7 to 6 because of his concern of what would happen if there are 10 jobs in one scanner type | 6 |
| 5 | Back-End Engineer/ Developer | Understands that lack of icon = didn't run; note: was the only participant to click on the pipeline number rather than the job failure to figure out why it failed; all links met his expectations of what he would want to see; describes design as clean, clear, has all the things he needs | Initially confused if red "x" means "didn't run" or "fail" but figured it out really quickly (and prototype doesn't include tooltips that would clarify this when deployed in production); also, same situation as participant 4: interpreted my scenario saying to look at projects where SAST "didn't run" as "where SAST was skipped", so tried to select "skipped" instead of "not found" in the filter dropdown, but when he saw that "skipped" wasn't enabled in the prototype he thought about it more and then chose "not found" - says he thinks the wording of "not found" is confusing; doesn't know if it was enabled or disabled or why it didn't run (if it was intentional or not) | 6 |
Note: we ran the test with a few more participants but the 5 above were the most qualified and didn't have any technical issues or language issues like some others did
Edited by Becka Lippert