Open issues based on Prometheus alerts
Many organizations use Issues for tracking incidents, like GitLab. Currently users have to manually create an issue, label it in some way as an incident, mention the proper folks, and then communicate that issue out to other channels.
This is very inconvenient, and we should automate this process.
Proposal
We can automate these tasks relatively simply, by automatically opening an issue when an alert is received from Prometheus.
The configuration can start off being very simple, with simply a checkbox to enable this functionality, which should default to on.
- New settings section
Operations -> Incidents
, this should be the top section in Operations - Short text blurb describing what this does
- Checkbox to enable automatic issue creation
- Checkbox to enable email messages sent to developers (current default)
The contents of the issue can then be:
For GitLab alerts: labels/gitlab_alert_id
-
title
- Metric title -
metric_query
- Configured metric query -
environment_name
- associated deployment environment
For external alerts:
- title -
annotations/title
orannotations/summary
- If it does not exist, build title out of
<receiver>: <startsAt>
just to have something that works out of the box
- If it does not exist, build title out of
- description -
annotations/descriptions
- Also include any other annotations included as well, for example
severity
,runbook
, etc. We can do special things with these later
- Also include any other annotations included as well, for example
Design
WIP screenshot
Documentation
There are a few places we should add documentation:
- We should create a new section in Settings -> Operations, which includes instructions for how to create issues from alerts
- We should include instructions for how to set this up for managed Prometheus instances, which is essentially no further action
- We should also document how to set this up for external Prometheus instances, as well as example configuration
- We should crosslink here from the Prometheus integration section
Edited by SV