Open issues based on Prometheus alerts

Many organizations use Issues for tracking incidents, like GitLab. Currently users have to manually create an issue, label it in some way as an incident, mention the proper folks, and then communicate that issue out to other channels.

This is very inconvenient, and we should automate this process.

Proposal

We can automate these tasks relatively simply, by automatically opening an issue when an alert is received from Prometheus.

The configuration can start off being very simple, with simply a checkbox to enable this functionality, which should default to on.

• New settings section Operations -> Incidents, this should be the top section in Operations
• Short text blurb describing what this does
• Checkbox to enable automatic issue creation
• Checkbox to enable email messages sent to developers (current default)

The contents of the issue can then be:

For GitLab alerts: labels/gitlab_alert_id

• title - Metric title
• metric_query - Configured metric query
• environment_name - associated deployment environment

For external alerts:

• title - annotations/title or annotations/summary
  • If it does not exist, build title out of <receiver>: <startsAt> just to have something that works out of the box
• description - annotations/descriptions
  • Also include any other annotations included as well, for example severity, runbook, etc. We can do special things with these later

Design

WIP screenshot

Documentation

There are a few places we should add documentation:

• We should create a new section in Settings -> Operations, which includes instructions for how to create issues from alerts
  • We should include instructions for how to set this up for managed Prometheus instances, which is essentially no further action
  • We should also document how to set this up for external Prometheus instances, as well as example configuration
• We should crosslink here from the Prometheus integration section