Indexing alerts should check pause indexing setting
Background
Advanced search indexing can be paused manually for Elasticsearch upgrades, shard size changes, reindexing, or other maintenance tasks.
Advanced search indexing can be paused automatically by Advanced search migrations.
New prometheus metrics were introduced in gitlab-org/gitlab!161258 (merged) for advanced search. The new metrics expose the boolean settings for advanced search (including whether indexing is paused or not) as a gauge where 1 = true and 0 = false
Our team gets alerted when known work is ongoing and we've forgotten to silence alerts. It would be a QOL improvement to allow the alerts to be aware of indexing being paused.
What does this MR
This MR does a few things:
- do not alert for the following alerts (copied from one of our change request templates) when advanced search indexing is paused:
alertname="SearchServiceElasticsearchIndexingTrafficAbsent"alertname="gitlab_search_indexing_queue_backing_up"alertname="SidekiqServiceGlobalSearchIndexingApdexSLOViolation"alertname="SidekiqServiceGlobalSearchIndexingTrafficCessation"
- reduce the staging threshold to
10_000, staging is much less busy than production and I would rather be alerted sooner for backups
Edited by Terri Chu