Label and annotate issues using GFM in alerts from external Prometheus instances

Problem to solve

Triaging of incidents takes time away from fire-fighting and can extend the resolution process of a single incident. We enable our users to be proactive by labelling incidents and inserting annotations automatically using GFM on incoming alert attributes. This lets them organize and categorize incidents automatically, minimizing time spent trying to figure who's board it needs to show up on.

Intended users

Sasha the Software Developer
Devon the DevOps Engineer
Sidney the Systems Administrator

Further details

This work contributes to the Incident Management Vision

Proposal

In the following proposal, the term gitlab_alert_template is used. This is simply for explanation purposes and we collaborate on the best name for this annotation.

User Journey

1. User reads documentation on how to use gitlab_alert_template and configures an alert in Prometheus utilizing this attribute. They add a block of text and some quick actions.
2. User creates a new issue template in GitLab and calls it Incident.
3. User selects the Incident issue template in Operations Settings so that when the Alert Bot creates issues automatically for alerts, these issues use the Incident issue template
4. Alerts start coming in and issues are automatically created. These issues display:
   • gitlab_alert_template annotation
   • bulleted list of other attributes
   • content User configured in the Incident issues template

Implementation Steps

1. We update code to expect the gitlab_alert_template annotation and insert it at the top of whatever issue template the user selects for Incident management.
2. We update the docs explaining to the user how to configure this annotation in Prometheus

What does success look like, and how can we measure that?

• % of issues created where gitlab_alert_template != NULL by instance using Prometheus alerting

Links / references