Skip to content

Draft: Add extra label to paging alerts

Craig Furman requested to merge craigf/extra-paging-label into master

The first step in a series of changes that will allow us to page and create issues from the same alert.

Currently, we use the pager label as a mutually exclusive switch between pagerduty and GitLab issues. This commit introduces a second label, page: 'true' to all current pager: 'pagerduty' alerts.

We enforce the use of this label in the validate-alerts test script.

After this is rolled out, a subsequent change would switch to using page: 'true' as the routing criteria for Alertmanager-Pagerduty routing.

We could then perform a similar pair of changes to change how issues are routed. First, add a label issue: <project> to every pager: 'issue' alert, where "project" is the value that is currently in the "project" label. Then, change the routing tree to use this new "issue" label to route alerts to issues. Finally, we can remove the "project" and "pager" labels for all alerts.

Related: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11571#note_434341653

This is a proposal, do not merge yet! I'm asking for review of the general idea as much as I'm asking for review of the code itself.

We need simultaneous issue and pagerduty routing for https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11571.

Merge request reports