Skip to content

Move logging and elasticsearch alerts to #g_infra_observability_alerts.

We presently have a number of alerts set up for logging clusters:

https://gitlab.com/gitlab-com/runbooks/-/blob/master/rules/logging.yml?ref_type=heads https://gitlab.com/gitlab-com/runbooks/-/blob/master/rules/elastic-clusters.yml?ref_type=heads https://gitlab.com/gitlab-com/runbooks/-/blob/master/mimir-rules/alerts/logging.yml?ref_type=heads https://gitlab.com/gitlab-com/runbooks/-/blob/master/mimir-rules/alerts/elastic-clusters.yml?ref_type=heads

They go to the default location, so #alerts. We are 100% not looking at these.

We probably should be for reasons like this: production-engineering#25105

This issue is to do the following:

  1. Delete the files in rules. We might as well just move this to Mimir while we're at it.
  2. Verify Mimir has all the data in question. A quick glance at the various alerts say they're more or less useful, so we should just move them and then deal afterwards.
  3. Update the mimir files to include team: scalability-observability and change https://gitlab.com/gitlab-com/runbooks/-/blob/master/services/teams.yml?ref_type=heads#L128-143 because I'm pretty sure the : isn't working as expected.
  4. Make sure things start alerting in #g_infra_observability_alerts.

I will do this after Summit.

Edited by Stephanie Jackson