Design: Impact Assessment Brainstorm

Problem and scope

See details

What is the problem to solve?

Security engineers need to understand the potential impact of security policies before enabling them to avoid disrupting development teams. Currently, validating policy impact requires creating test projects, trialing with subsets, and other manual, time-consuming tasks. This leads to either overly cautious policy rollouts or unexpected disruptions that damage trust between security and development teams.

Who is the design solution for?

Primary: Alex (Security Operations Engineer)

Secondary: Sam (Security Analyst)

What is the Job this user is trying to achieve?

  • Configure and enable security policies that protect the organization without unexpectedly blocking legitimate development work
  • Build trust with development teams by demonstrating thoughtful, data-driven policy implementation
  • Validate policy effectiveness before full enforcement
  • Identify teams that need training or support before policy enforcement

What outcomes is this design solution helping them achieve?

If you have measured Outcome data, put that in the table. If not, delete the table and add the Outcomes to be designed for in a bulleted list.

  • Reduce the number of development disruptions caused by new security policies
  • Minimize the time it takes to validate a policy's impact before enabling it
  • Increase confidence in policy decisions through data-driven insights
  • Identify which teams need preparation or training before policy enforcement

What are the requirements necessary to solve for this problem and Outcomes?

{ Create a bulleted list of everything that will need to be addressed in the holistic Design Vision in order to fully solve for the problem and Outcomes. Work with your counterparts to define this list. Keep it solution agnostic and try to understand any technical constraints that each requirement may imply. Only remove a requirement due to a technical constraint if it's not technically feasible to build within a reasonable amount of time (1-3 milestones is reasonable, anything longer than that you'll have to decide as a team how important it is to keep it in scope or address it in parallel in later iterations). }

  • Impact Analysis Engine
    • Ability to analyze historical data (last 30 days minimum) of merge requests
    • Support for "what-if" analysis based on new or modified policy rules
    • Must work with existing scan results and merge request data
  • Visualization & Reporting
    • Clear visualization of potential blocks/violations
    • Team-by-team impact breakdown
    • Identification of specific violations and their frequency
    • Exportable reports for stakeholder communication
  • Performance Requirements
    • Analysis must complete within reasonable time (< 2 minutes for 30-day analysis)
    • Must handle organizations with thousands of MRs per month
  • Integration Points
    • Seamless integration with policy creation/editing workflow
    • Access to historical merge request and scan data
    • Connection to team/project ownership data
  • Action Enablement
    • Direct links to affected MRs/projects
    • Ability to adjust policy rules based on findings
    • Options to notify affected teams
    • Integration with training/documentation systems

What supporting research or customer validation do we have?

  • { Add links to any supporting research and related problem validation issues }

What is the timeline?

{ Add milestone or link to planning issue that clarifies when the design must be ready for the Build phase by }

18.3

What are the technical constraints?

{ ⚠️ This is to understand initial constraints in which the design solution needs to work within, NOT whether the solution can be implemented in a given milestone.

Once the Product Designer has come up with a holistic Design Vision, or an ideal state for solving the problem, they should collaborate with their team members and engineers to continue the technical feasibility discussion during workflowplanning breakdown. }

  • { e.g. All visualization must use [eCharts](https://echarts.apache.org/examples/en/index.html#chart-type-line) }
  • { e.g. Data prior to 2024-10-10 will not be available }
  • { e.g. Solution will only be visible to Maintainer roles }

In what parts of GitLab will this solution be available?

Plans:

  • Free
  • Premium
  • Ultimate

Instances:

  • Self-managed
  • Dedicated
  • GitLab.com

Levels:

  • Instance
  • Group
  • Project

How will we know if the solution is successful?

{ If you don't have measured Outcome data to measure against, what other success metrics can we use? Otherwise you can reference the Outcomes table above (your goal is to beat the previous Outcome measurements Satisfaction numbers with this new design). }

  • Impact Reduction Metrics
    • Unexpected Disruption Events: Number of policies disabled within 24 hours of enablement due to unexpected impact (target: <5%)
    • Developer Complaints: Support tickets related to "surprise" policy blocks (target: reduce by 75%)
    • Policy Revision Rate: % of policies modified after seeing impact assessment (shows proactive adjustment)
  • Adoption Metrics
    • Impact Assessment Usage Rate: % of new policies that run impact assessment before enablement (target: >80% within 3 months)
    • Time to Policy Enablement: Average time from policy creation to enforcement (target: reduce by 50%)
    • Assessment Completion Rate: % of users who complete the full assessment flow vs. abandoning (target: >90%)
  • Efficiency Metrics
    • Policy Testing Time: Hours spent on manual policy validation (target: reduce by x%)
    • False Positive Rate: % of blocked MRs that required exception/override (target: < y%)
    • Time to Identify Impact: How quickly teams can understand policy implications (target: < 'z' minutes per assessment)

Ready for design

See checklist
  • The problem has been defined and is well understood
  • Who the design solution is for has been defined
  • User goals and outcomes have been defined
  • Supporting research has been reviewed and linked
  • The product requirements have been defined, and the scope has been agreed upon
  • Success metrics have been defined and agreed upon
  • I, as the Product Designer, believe I have all the information I need to begin creating a design solution
  • Move this issue to ~"workflow::ready for design" or workflowdesign 🎉
  • (Optional) Help improve this issue template, view feedback issue

Proposal

See checklist reminder

Design breakdown

Once the proposal is agreed upon, work with your team to break it down into buildable parts (MVC, Iteration 1, Iteration 2, etc... until fully built).

See checklist
  • Design Vision broken down into MVC and follow-up iterations based on their ability to stand alone and provide value to the user
  • Create MVC and other necessary Iteration 1, Iteration 2... issues and add them as Linked items to this issue
    • Include all necessary requirements, and specs needed to create the designs for each broken down issue
Edited by Torian Parker