Solution Validation: DAST On-demand Configuration Improvements

What did we learn?

Summary of Results
The proposed changes to the DAST on-demand configuration UI received a lot of positive feedback from study participants. The updated interface design was easy to use and the additional UI copy made the experience simpler to learn compared to the existing UI. While the feedback was generally positive and validates the design direction, a handful of opportunities and pain points were uncovered that should be addressed in future iterations.
https://dovetailapp.com/projects/7Mcg4vwhxzT3M0tg6vgAVM/insights/present
Summary of findings
  1. Overall, the experience received high ratings from participants (4.5/5)
  2. The updated page design and supplemental UI text improves learnability & comprehension
  3. Most participants were able to understand what profiles are and how they should be used based on the information provided
  4. Participants viewed their experience interacting with the DAST profiles UI as “easy”, “intuitive”, and in line with their expectations
  5. Some participants expected to find "pre-build" profiles
  6. Summary of pain points & opportunities
Actionable Insights
# Insight Action Issue
1 Users found the “edit” and “change” profile actions to be ambiguous and redundant. Iterate on the UI to differentiate these actions or consider removing the “edit” action on the selected card. DAST: Differentiate between the "edit" and "cha... (gitlab#356418)
2 The additional UI text was helpful to many, but some participants still have problems understanding some aspects of the interface. Iterate on the UI to address some of the learnability issues that users encountered. DAST: Iterate on the configuration UI to furthe... (gitlab#356421)
3 A number of users expected to find pre-build profiles they could use. Explore adding pre-build scanner profiles to every project. DAST: Add pre-built scanner profiles (gitlab#356439)
4 Users misinterpret the “Active” scan mode value, thinking it represents that the scan is enabled. Explore ways to minimize the point of confusion DAST: Explore ways to clarify "Active" and "Pas... (gitlab#356441)
5 Users were confused why they couldn’t “view results” of a running, canceled, or failed scan. Explore possibilities of adding a “view results” button for any on-demand scan. Where could it lead for each scan state? On-demand scans: Add "View Results" action to s... (gitlab#356446)

What's this issue all about? (Background and context)

This issue is to validate the design solution proposed in 🎨 Design: On-demand DAST Configuration Improvem... (gitlab#351476 - closed)

What are the overarching goals for the research?

Validate that the proposed design solution improves task success and identify any potential pain points or usability issues prior to implementation planning.

What hypotheses and/or assumptions do you have?

We believe that the repeated context switching required to enable DAST is a pain point for many users. Furthermore, portions of the current configuration UI are not clear or easy for some to understand. By minimizing the context switching within the configuration workflow and adding copy to better explain the high-level concepts and available options, we will improve user comprehension and task success compared to the current implementation.

What research questions are you trying to answer?

  • Are users who are not familiar with DAST able to complete the configuration workflow and run an on-demand scan?
    • Do they feel successful?
    • Do they feel informed? (i.e. are they able to figure out what they are doing?)
  • Does the approach to reduce context switching within the configuration workflow create a favorable experience for users?
    • Does the profile selection and management process meet users' expectations?
    • Should we consider any alternative patterns?
    • Do users understand the relationship between profiles and DAST?
  • Are users able to differentiate between DAST CI/CD and DAST on-demand scanning?
  • Do the UI changes that make the on-demand experience less specific to DAST create any pain points or usability issues?

What persona, persona segment, or customer type experiences the problem most acutely?

  1. Sasha (Software Developer)
  2. Sam (Security Analyst)

What business decisions will be made based on this information?

We will validate the direction of the proposed design updates to configure a DAST on-demand scan and inform product direction going forward.

What, if any, relevant prior research already exists?

Who will be leading the research?

@mfangman will be conducting unmoderated usability testing on usertesting.com

What timescales do you have in mind for the research?

Research should be completed by the end of %14.9

Relevant links (script, prototype, notes, etc.)

Edited by Michael Fangman