## Problem to be solved Today, Incident management is set up to trigger escalation policies for new alerts. In this scenario, the on-call responder who is paged can end the paging by acknowledging the alert by changing it's status to from triggered to acknowledged. If the responder changes the status back, we restart the escalation policy and begin paging again. When a user creates an incident manually, there is no associated alert. We need to enable paging on incidents and the ability for a responder to "acknowledge" and end paging for a manually created incident AND to "un-acknowledge" or restart paging on an incident for a different escalation policy or user. ## Intended Users * [Devon (DevOps Engineer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#devon-devops-engineer) * [Allison (Application Ops)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#allison-application-ops) * [Priyanka (Platform Engineer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#priyanka-platform-engineer) ## User Experience User creates an incident and selects the Escalation Policy or user to trigger paging for that Incident. On-call responder can "acknowledge" an incident to indicate that they are working on the incident and to end paging. ## Design Introduce sidebar items that surface Status and Escalation Policies: | Status and escalation policy sidebar items | Status dropdown options | More information about what changing the status does | Escalation policy dropdown | No escalation policies created | No escalation policies created - expanded | Hidden edit button for those without proper permissions | System note for Paging status changes | Surface status on incident list | | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | |  | | |  |  |  | | | | Notes: - Only `developers and up` will be able to edit the incident status or escalation policy. Reporters and non-project members will have the `Edit` button hidden. - Changing the incident status to `acknowledged` or `resolved` will stop paging according to the specified escalation policy. On the other hand, changing the status from `acknowledged` or `resolved` to `triggered` will re-start paging. - If an incident was created from an alert, the alert and incident statuses will be mirrored (so, an ACKed alert will become and ACKed incident). - For incidents created from alerts - if an escalation policy has been created for the project, the escalation policy will be pre-populated when the incident is created. For manual incidents, the escalation policy needs to be defined manually. Email to users when paged: > An incident has been triggered in \[group/project\]. > > [View incident details](/gitlab-org/gitlab/-/blob/master/url) > > Title: \[Insert title here\] > > Description: \[Insert description here\] > > Escalation policy: \[Insert escalation policy, if present\] > > Metric: \[Insert metric, if available\]* _[Metric could be a string or a link, up to the discretion of the engineer implementing this issue. Longer-term, we'll likely include a png of the metric but that's out of scope for the first iteration.]_ [Figma file](https://www.figma.com/file/EGigpP5ioLGPolJolNvjze/Ack-ing-and-escalating-manually-created-incidents-gitlab-com-and-323367?node-id=63%3A342) ### Issue Flow ```mermaid graph LR tables("Add base tables/models [BE]"):::has-tooltip click tables href "https://gitlab.com/gitlab-org/gitlab/-/issues/330277" "Add tables/models for IssuableEscalationStatus & IssuableEscalations" statusview("Add status dropdown [BE, FE]"):::has-tooltip click statusview href "https://gitlab.com/gitlab-org/gitlab/-/issues/330283" "Add a 'Status' dropdown in the UI on the incident page" statuslist("Show status in incident list [FE]"):::has-tooltip click statuslist href "https://gitlab.com/gitlab-org/gitlab/-/issues/332056" "Surface paging status on incident list" policydropdown("Add policy dropdown [BE, FE]"):::has-tooltip click policydropdown href "https://gitlab.com/gitlab-org/gitlab/-/issues/330284" "Add Escalation policy dropdown in the UI" statusupdates("Propagate status changes to escalations [BE]"):::has-tooltip click statusupdates href "https://gitlab.com/gitlab-org/gitlab/-/issues/332065" "Update escalations when incident status is changed" slash("Add slash command [BE]"):::has-tooltip click slash href "https://gitlab.com/gitlab-org/gitlab/-/issues/330282" "Add slash command for escalating an incident" escalate("Start escalating [BE]"):::has-tooltip click escalate href "https://gitlab.com/gitlab-org/gitlab/-/issues/330281" "Escalate manually created incidents according to an escalation policy" systemnotes("Add system notes [BE]"):::has-tooltip click systemnotes href "https://gitlab.com/gitlab-org/gitlab/-/issues/332055" "Add system note for paging status changes" resolve_on_close("Resolve incident when closed [BE]"):::has-tooltip click resolve_on_close href "https://gitlab.com/gitlab-org/gitlab/-/issues/334713" "Resolve status on manually created incidents when incident is closed" tables --> statusview statusview-->statuslist statusview-->statusupdates statusview-->systemnotes statusview-->resolve_on_close tables-->policydropdown policydropdown --> systemnotes policydropdown === slash tables-->escalate escalate-- partially blocking -->policydropdown escalate-->statusupdates ```