MVC of SLI Alerts
As a first step towards https://gitlab.com/gitlab-org/gitlab-ee/issues/4451, we should provide the absolute minimum features for alerting: setting an alert, receiving a notification, and seeing what the value is.
I believe the smallest iteration is:
- Allow configuration of a single alert threshold, for each chart.
- The only configurable options are: alerting threshold, operator (
>
,<
,=
) - The alert icon would not change on a triggered alert, it simply exposes the drop down to edit the values.
- Available to Masters and Owners only.
- The only configurable options are: alerting threshold, operator (
- A red horizontal line at the alerting threshold, displayed on the chart. (Stretch)
- An email sent to project Masters and Owners, in the event an alert triggers. Email is sent based on Project Notification settings. (No email on clear, right now.)
Adding an alert
To trim this down for the MVC, we should:
- Pop this menu directly when you click on the alert icon
- Change
Add
toSave
- Skip the
duration
field.
Designs
- When you click the
notification-off
button, a 'New alert' dropdown shows up, and you can add an alert (Mockup 01) - After clicking 'Save' button in the 'New alert' dropdown, the
notification-off
icon will change tonotification-on
icon. And it shows150 > Threshold
text next to the notification button. (Mockup 02) - You can delete the alert by clicking 'Delete' button in the dropdown (Mockup 02). However, if you change the settings of the alert, the 'Delete' button will change to 'Save' button (Mockup 03).
- If you delete the alert, the '150 > Threshold' text will disappear, and the
notification-on
icon will change tonotification-off
icon at the same time.
| 01 - Add alert | 02 - Edit alert -- Delete | 03 - Edit alert -- Save | | --- | --- | --- | --- | | | | |
Alert Email
We should implement a web hook receiver for Prometheus' alert manager to trigger. Based on the alert, we should email Masters and Owners of the project with an email:
Subject: {Project} | Alert: {Environment} - {Metric} {operator} {threshold} for 5 minutes
Body:
An alert has been triggered in {Project}.
Environment: {environment}
Metric: {Metric} {operator} {threshold}
[View {environment} performance dashboard]().
--
You're receiving this email because of your account on [gitlab.com]. If you'd like to receive fewer emails, you can [adjust the alert](link to performance dashboard) or your [notification settings](https://gitlab.com/profile/notifications).
Setting alerts in AlertManager
In order for Prometheus to generate alerts, we need to tell it what to look for. To accomplish this we should:
- Enable Alert Manager in the Prometheus chart
- Utilize
helm upgrade --reuse-values
to pass the rule and alert configuration based on the configured alerts.
Edited by Joshua Lambert