Move logging and elasticsearch alerts to #g_infra_observability_alerts.
We presently have a number of alerts set up for logging clusters:
https://gitlab.com/gitlab-com/runbooks/-/blob/master/rules/logging.yml?ref_type=heads https://gitlab.com/gitlab-com/runbooks/-/blob/master/rules/elastic-clusters.yml?ref_type=heads https://gitlab.com/gitlab-com/runbooks/-/blob/master/mimir-rules/alerts/logging.yml?ref_type=heads https://gitlab.com/gitlab-com/runbooks/-/blob/master/mimir-rules/alerts/elastic-clusters.yml?ref_type=heads
They go to the default location, so #alerts. We are 100% not looking at these.
We probably should be for reasons like this: production-engineering#25105 (moved)
This issue is to do the following:
- Delete the files in rules. We might as well just move this to Mimir while we're at it.
- Verify Mimir has all the data in question. A quick glance at the various alerts say they're more or less useful, so we should just move them and then deal afterwards.
- Update the mimir files to include team: scalability-observability and change https://gitlab.com/gitlab-com/runbooks/-/blob/master/services/teams.yml?ref_type=heads#L128-143 because I'm pretty sure the : isn't working as expected.
- Make sure things start alerting in #g_infra_observability_alerts.
I will do this after Summit.
Edited by Stephanie Jackson