BUSINESS CASE: Incidents require a specific UI/UX

Executive Summary

This is a proposal to create a new UI/UX for Incidents that is unique. Currently, incidents are generic GitLab issues with the incident label. This has proved to be insufficient because incident management tools need to be streamlined, intuitive, and uncluttered as they are leveraged during high-stress scenarios when time is money. If GitLab wants to be competitive in the Incident Management market, we need to build a tool starring Incidents.

Overview

Background

GitLab Issues, as a small primitive, have been applied for many different use cases: Development work, support ticket, incidents, task, security incident etc. Issues have a single set of features that are available for all of the use cases. Whenever a feature is added to issues, it is visible and available for all use cases. Using GitLab Issues helped to get Incident Management quickly off the ground. But as we continue to invest in Incident Management, it has become restrictive for the Operations use case with regards to IT Incidents. There are many changes that would enhance the incident management experience for a responder (see below), but since those changes would impact everyone using issues for all use cases, we cannot make those improvements when incidents are just a generic issue. Furthermore, there are potential security and access considerations that make managing an incident a separate concern from managing a generic issue.

Why is this important and why do we need to do it now?

In 2 year’s time, GitLab aims to make observability a commodity by being ubiquitous, complete, cost effective, and simple to setup and operate for any cloud-native team, enabling them to continuously improve. By using GitLab, teams can reduce the frequency and severity of incidents in production (this vision statement has been pulled from GitLab's Monitor Direction).

In order for us to build a tool that enables cloud-native teams to continuously improve their systems, applications, and team processes, we need an intuitive user interface coupled with an incredible user experience for people responding and remediating problems in production. If we are constrained to using generic issues as incidents, we will not be able to create a stream-lined experience whereby responders can triage alerts and remediate incidents with ease.

We need to make this change now because we are heavily investing in Alert and Incident Management in the Monitor stage. Our current designs are centered around issues as incidents, and the constraints that come along with generic issues. If we are to change this paradigm and create a new UI/UX for incidents we need to make this change now so that we avoid more re-work for design and development.

Opportunities

The following features/experiences will be unblocked if we create a new UI for Incidents:

• Well-organized schema of alert payload, metrics, logs, and incident timeline. We want to make it easy to take notes or add an event to the timeline WHILE being able to view metrics & logs (if all of this information is in the same mark-up field this is impossible)
• Multiple people can edit aspects of the incident at the same time (e.g. add context to the description AND add an event to the incident timeline)
• Customization of the side bar
  • Removal of 6 side bar features that are not related to incident management
  • Embedded collaboration mechanisms
  • Addition of incident management actions: Acknowledge, Dismiss, Resolve
  • Status Page easy access
  • Access control specific to incidents
• Dynamic and embedded alert information (payload & event count) that updates automatically

Here is a vision mock for inspiration:

Pros & Cons

Using generic issues for incidents

Pros

• Issues already exist and are a known paradigm in GitLab

Cons

• It is very easy for someone can simply remove or add the incident label on an issue with zero awareness of the rippling consequences making it difficult to do the following:
  • Restrict access and control permissions - Alerts & incidents contain highly sensitive information that will require restricted access. Currently, there is no known pattern for restricting access to Issues based on labels.
  • Create a triage board for incidents - There is a specific workflow for incidents that is different than issues. Simply using incident boards is inadequate.
  • Distinguish between an issue and an incident - Accidentally removing the incident label can remove the incident from critical workflows. This is a huge risk to users.
  • Reporting on incidents - Harder to do data reporting based on incidents because the only designation is a label that can be easily added or removed (e.g. over the past year, we had 14 severe incidents that took an average of 18 min to resolve)
• Editing the description of an issue with well-formatted markdown is cumbersome and slow. This is problematic during a firefight, made more so if there is a large alert payload, embedded metrics, embedded logs, etc
  • There can be a lot of information that are important to different users. Having most of the information in the description is not ideal because the information can be hard to find. This problem is made worse when people do not want to spend their time writing well-formatted markdown.
• The UI of Issues cannot be customized for Incident Management - There is a lot of extra stuff on issues that clutters an incident. Incidents need to be easy to quickly parse visually. Issues contain many features that are not useful for incidents and distract from the important information: epic, milestone, time tracking, due date, weight, health status
• The issue description is a single field that only one person can edit at a time - This is why the SRE team uses google docs. We can solve this by creating separate fields for payload, notes, timeline, etc

Incidents established separately to Issues

Pros

• Build a streamlined triage experience - Incidents are different than issues. Incidents are used in high-stakes stressful situations so the experience needs to be different and more streamlined than managing a backlog of issues
• Add reporting for incident frequency, MTTA/MTTR - We will be able to identify and track incidents in someone's GitLab instance and offer robust reports with sound data. Removing a label from an incident will not adversely affect the foundation of the reports
• Establish clear permissions around what roles have access to alerts and incident - It will be easy to create a permission structure for incidents as a dedicated object.
• Establishing alerts, incidents, and issues as different entities is a known pattern in all other incident management tools - In PagerDuty, OpsGenie, and VictorOps incidents are critical alerts. Action items that come from post incident reviews are captured as tickets in a workflow tool (JIRA, ServiceNow, GitLab)

Cons

• The Health group has added features to issues that may need to be rebuilt (Embedded metrics, zoom quick actions, slack slash commands)
• Status Page is based on issues - this would need to be reworked. HOWEVER, it is still useful and enables the SRE team to move to ops.gitlab.net so while we will need to rework Status page in the future, it still serves an important purpose right now. ALSO, based on a high-level engineering assessment, this will not be difficult.
• We may need to build functionality into alerts & incidents that we lose by not using issues such as: assignee, comments, labels
• We may need a migration plan if we find that any customer is using the auto-create issue endpoint which we need to deprecate
• Future improvements relevant to incidents won't automatically be included when GitLab improves issues

Solutions & Scope of Work

Incidents need a unique UI/UX. The how is ultimately up to engineering. Some initial thoughts on how we might to this:

1. Establish Incidents as a new object. We estimate that it will take approximately 3 months to re-add the following features to Incidents that we lose from issues:

• Add the new database table
• Instantiate incident lists and detail page in the UI
• Add comment and polling (i.e. real time updates to the page)
• Add assignees
• Add relationship to MRs and ability to create an MR from an incident

2. Explore creating Issue Types. Discussions are on-going with the Plan team. This issue will be updated when/if there is an opportunity to pursue it.