Automatically embed metrics in issue for all gitlab-configured alerts
Problem to solve
Charts help users visualize what went wrong with triaging an incident. If a particular threshold was exceeded, we can reduce time spent during investigation by automatically embedding the relevant metrics chart in the issue.
Intended users
Sasha the Software Developer
Devon the DevOps Engineer
Sidney the Systems Administrator
Further details
This work contributes to the Incident Management Vision
Proposal
The first iteration will involve metrics for GitLab-configured alerts (where a chart already exists on our metrics dashboard). We would:
- Find the metric and use the corresponding dashboard/embed configuration including chart title, units, etc.
- Automatically embed a chart visualization for the metric that triggered the alert. Set the time frame to be event time +/- 30 minutes.
The chart should display in the issue description, after the summary and before the alert details (ie, the rest of the alert payload).
Out of scope for this iteration
- Embedding metrics in issue from Prometheus alerts not created in GitLab. Will be tackled in #195739 (closed)
- Embedding metrics in issues created from alerts triggered by the generic alert endpoint