feat: have api inhibit alerts from patroni
What
When the patroni
service is violating SLO, stop the api
service from
alerting.
To look at the generated configuration run ./alertmanager/generate.sh which will add the following:
alertmanager configuration
- equal:
- env
- environment
- pager
source_matchers:
- component="rails_primary_sql"
- type="patroni"
target_matchers:
- component="graphql_queries"
- type="api"
- equal:
- env
- environment
- pager
source_matchers:
- component="rails_replica_sql"
- type="patroni"
target_matchers:
- component="graphql_queries"
- type="api"
- equal:
- env
- environment
- pager
source_matchers:
- component="rails_primary_sql"
- type="patroni"
target_matchers:
- component="loadbalancer"
- type="api"
- equal:
- env
- environment
- pager
source_matchers:
- component="rails_replica_sql"
- type="patroni"
target_matchers:
- component="loadbalancer"
- type="api"
- equal:
- env
- environment
- pager
source_matchers:
- component="rails_primary_sql"
- type="patroni"
target_matchers:
- component="nginx_ingress"
- type="api"
- equal:
- env
- environment
- pager
source_matchers:
- component="rails_replica_sql"
- type="patroni"
target_matchers:
- component="nginx_ingress"
- type="api"
- equal:
- env
- environment
- pager
source_matchers:
- component="rails_primary_sql"
- type="patroni"
target_matchers:
- component="puma"
- type="api"
- equal:
- env
- environment
- pager
source_matchers:
- component="rails_replica_sql"
- type="patroni"
target_matchers:
- component="puma"
- type="api"
- equal:
- env
- environment
- pager
source_matchers:
- component="rails_primary_sql"
- type="patroni"
target_matchers:
- component="rails_requests"
- type="api"
- equal:
- env
- environment
- pager
source_matchers:
- component="rails_replica_sql"
- type="patroni"
target_matchers:
- component="rails_requests"
- type="api"
- equal:
- env
- environment
- pager
source_matchers:
- component="rails_primary_sql"
- type="patroni"
target_matchers:
- component="workhorse"
- type="api"
- equal:
- env
- environment
- pager
source_matchers:
- component="rails_replica_sql"
- type="patroni"
target_matchers:
- component="workhorse"
- type="api"
Affected alerts
Reference: https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/15767
Edited by Steve Xuereb