Solution Validation for gitlab#466441

What did we learn?

Results
2-3 sentences to summarize the results
- Link to Dovetail project

What's this issue all about? (Background and context)

Conducting usability testing on the new concept for a Secret Detection Allowlist. Design issue: Design: Secret Detection Exclusions - Vision (gitlab#466441 - closed)

What are the overarching goals for the research?

  1. Understand what is and is not working well about the concept. Identify areas for improvement and/or refinement.
  2. Determine if users have a preference for scanner-specific allowlists or a unified allowlist.
  3. Generate a UXUM-Lite score and corollary SUS score. Compare scores with usability benchmarking study.

What hypotheses and/or assumptions do you have?

  • User Understanding of Allowlists: Many users, especially those less familiar with security or GitLab, may not fully understand the concept of an allowlist and its purpose within the application.
  • Allowlist Navigation Success: Users who have a grasp of the allowlist concept will generally be able to navigate the prototype effectively.
  • "Inherited from" Confusion: Users unfamiliar with GitLab's group/project structure may find the "Inherited from" content confusing.
  • Allowlist Item Addition: Adding an allowlist item should be a relatively straightforward process for most users.
  • Scanner-Specific Allowlists Preference: Users will likely prefer to manage allowlists at the scanner level, rather than having a single global list, to maintain better control over their security configurations.

What research questions are you trying to answer?

Understanding and Navigation
  1. UI Comprehension: Are users able to understand the basic concepts and terminology related to allowlists within the provided UI?
  2. Navigation Efficiency: Can users easily locate and navigate to the allowlist settings or configuration options?
  3. Intuitive Interface: Is the allowlist UI intuitive and easy to use, with minimal confusion or frustration?
Scanner-Specific Allowlists
  1. Benefits Recognition: Do users understand the advantages of having scanner-specific allowlists, such as increased granularity and control over security settings?
  2. Value Perception: Do users perceive the benefits of scanner-specific allowlists as outweighing any potential drawbacks or increased complexity?
Granularity vs. Efficiency
  1. Trade-Off Assessment: How do users weigh the benefits of granularity in scanner-specific allowlists against the potential efficiency gains of a unified allowlist?
  2. Preference Determination: Do users have a strong preference for one approach over the other, or are their preferences more nuanced?
Allowlist Application
  1. Understanding of Scope: Do users understand how the allowlist applies to different levels of the hierarchy (project, group, etc.) and how it interacts with various secret detection methods?
  2. Expected Behavior: Do users' expectations regarding the application of allowlists align with the actual behavior of the system?

What persona, persona segment, or customer type experiences the problem most acutely?

  • Primary persona: Amy (Application Security Engineer)
    • JTD: As an AppSec practitioner, I want to eliminate specific secret detection findings, so that I reduce as much organizational friction as possible, and reduce time spent triaging unnecessary findings, while using secret detection to maintain good security controls.
  • Secondary persona: Sasha (Software Developer)
    • JTBD: As a developer, I want to reduce irrelevant secret detection findings, so that I can maintain an efficient workflow.

What business decisions will be made based on this research?

This research will help determine if the design direction for an Allowlist meets users or if changes are needed before implementation.

What would happen if we did not do this research? Please describe the risk to the GitLab and/or the Product Division.

  • We may end up building the allowlist in a way that doesn't align with user expectations.
  • We potentially add more complexity and/or confusion in the UI

What, if any, relevant prior research already exists?

  • None

Who will be leading the research?

What timescales do you have in mind for the research?

ASAP

What KR is connected to this research issue? If the KR is for a future quarter, elaborate on how it connects to company yearlies or supports product strategy at the section, stage, or stage group level.

  • None

Relevant links (script, prototype, notes, etc.)

Edited by Michael Fangman