Link alerts from GitLab-managed Prometheus to logs
Problem to solve
When triaging alerts, you need to be able to quickly investigate the issue, ideally without a lot of context switching, which costs you time that you can't afford to lose during an outage. By providing a link in the alert to directly the logs for a metric, GitLab saves you time by minimizing context switching. Logs can be accessed via the drop-down menu in the upper righthand corner of the chart in the metrics tab in an alert.
Intended users
Further details
This work supports the direction of the Alert Management product category.
Proposal
Link to logs for the metric threshold that has been exceeded. Currently these are accessible from the metrics dashboard on the specific chart.
Design
As a first iteration, we can make use of the existing "View logs" option in the dropdown of our metrics charts. This dropdown and link will only be available when logs for the given metrics already exist within GitLab. In a second iteration, we can surface logs directly within a separate tab in the alerts themselves:
| First iteration | Second iteration |
|---|---|
 |
 |
- Note that this functionality will primarily benefit those people utilizing our Prometheus service. We can introduce the link to logs as long as the logs already exist within Gitlab.
- When we get to the point of adding a new tab to display logs: for alerts that originate outside of GitLab, we could add a field to the payload of the alert called
log_urland a user could populate it if they want. We could then look for that link and surface it in thelogstab. Alternately, we could perhaps hide the link to the log tab for non-GitLab alerts.
Release Post Image
Permissions and Security
Documentation
Documentation required. Please add a new sub-section to this section.