Label and annotate issues using GFM in alerts from external Prometheus instances
Problem to solve
Triaging of incidents takes time away from fire-fighting and can extend the resolution process of a single incident. We enable our users to be proactive by labelling incidents and inserting annotations automatically using GFM on incoming alert attributes. This lets them organize and categorize incidents automatically, minimizing time spent trying to figure who's board it needs to show up on.
This work contributes to the Incident Management Vision
In the following proposal, the term
gitlab_alert_template is used. This is simply for explanation purposes and we collaborate on the best name for this annotation.
- User reads documentation on how to use
gitlab_alert_templateand configures an alert in Prometheus utilizing this attribute. They add a block of text and some quick actions.
- User creates a new issue template in GitLab and calls it
- User selects the
Incidentissue template in Operations Settings so that when the Alert Bot creates issues automatically for alerts, these issues use the
- Alerts start coming in and issues are automatically created. These issues display:
- bulleted list of other attributes
- content User configured in the
- We update code to expect the
gitlab_alert_templateannotation and insert it at the top of whatever issue template the user selects for Incident management.
- We update the docs explaining to the user how to configure this annotation in Prometheus
What does success look like, and how can we measure that?
- % of issues created where
gitlab_alert_template!= NULL by instance using Prometheus alerting