Skip to content

MVC of SLI Alerts

As a first step towards https://gitlab.com/gitlab-org/gitlab-ee/issues/4451, we should provide the absolute minimum features for alerting: setting an alert, receiving a notification, and seeing what the value is.

I believe the smallest iteration is:

  • Allow configuration of a single alert threshold, for each chart.
    • The only configurable options are: alerting threshold, operator (>,<,=)
    • The alert icon would not change on a triggered alert, it simply exposes the drop down to edit the values.
    • Available to Masters and Owners only.
  • A red horizontal line at the alerting threshold, displayed on the chart. (Stretch)
  • An email sent to project Masters and Owners, in the event an alert triggers. Email is sent based on Project Notification settings. (No email on clear, right now.)

Adding an alert

Add Alert

To trim this down for the MVC, we should:

  • Pop this menu directly when you click on the alert icon
  • Change Add to Save
  • Skip the duration field.

trim-down

Designs

  1. When you click the notification-off button, a 'New alert' dropdown shows up, and you can add an alert (Mockup 01)
  2. After clicking 'Save' button in the 'New alert' dropdown, the notification-off icon will change to notification-on icon. And it shows 150 > Threshold text next to the notification button. (Mockup 02)
  3. You can delete the alert by clicking 'Delete' button in the dropdown (Mockup 02). However, if you change the settings of the alert, the 'Delete' button will change to 'Save' button (Mockup 03).
  4. If you delete the alert, the '150 > Threshold' text will disappear, and the notification-on icon will change to notification-off icon at the same time.

| 01 - Add alert | 02 - Edit alert -- Delete | 03 - Edit alert -- Save | | --- | --- | --- | --- | | 01-alert--add | 02-alert--edit | 03-alert--edit |

00-alert

Alert Email

We should implement a web hook receiver for Prometheus' alert manager to trigger. Based on the alert, we should email Masters and Owners of the project with an email:

Subject: {Project} | Alert: {Environment} - {Metric} {operator} {threshold} for 5 minutes
Body: 
An alert has been triggered in {Project}.  
Environment: {environment}
Metric: {Metric} {operator} {threshold}

[View {environment} performance dashboard]().

--
You're receiving this email because of your account on [gitlab.com]. If you'd like to receive fewer emails, you can [adjust the alert](link to performance dashboard) or your [notification settings](https://gitlab.com/profile/notifications).

Setting alerts in AlertManager

In order for Prometheus to generate alerts, we need to tell it what to look for. To accomplish this we should:

  • Enable Alert Manager in the Prometheus chart
  • Utilize helm upgrade --reuse-values to pass the rule and alert configuration based on the configured alerts.
Edited by silv