Skip to content

feat: have api inhibit alerts from patroni

Steve Xuereb requested to merge feat/inhibit-api-alerts-from-patroni into master

What

When the patroni service is violating SLO, stop the api service from alerting.

To look at the generated configuration run ./alertmanager/generate.sh which will add the following:

alertmanager configuration 
  - equal:
      - env
      - environment
      - pager
    source_matchers:
      - component="rails_primary_sql"
      - type="patroni"
    target_matchers:
      - component="graphql_queries"
      - type="api"
  - equal:
      - env
      - environment
      - pager
    source_matchers:
      - component="rails_replica_sql"
      - type="patroni"
    target_matchers:
      - component="graphql_queries"
      - type="api"
  - equal:
      - env
      - environment
      - pager
    source_matchers:
      - component="rails_primary_sql"
      - type="patroni"
    target_matchers:
      - component="loadbalancer"
      - type="api"
  - equal:
      - env
      - environment
      - pager
    source_matchers:
      - component="rails_replica_sql"
      - type="patroni"
    target_matchers:
      - component="loadbalancer"
      - type="api"
  - equal:
      - env
      - environment
      - pager
    source_matchers:
      - component="rails_primary_sql"
      - type="patroni"
    target_matchers:
      - component="nginx_ingress"
      - type="api"
  - equal:
      - env
      - environment
      - pager
    source_matchers:
      - component="rails_replica_sql"
      - type="patroni"
    target_matchers:
      - component="nginx_ingress"
      - type="api"
  - equal:
      - env
      - environment
      - pager
    source_matchers:
      - component="rails_primary_sql"
      - type="patroni"
    target_matchers:
      - component="puma"
      - type="api"
  - equal:
      - env
      - environment
      - pager
    source_matchers:
      - component="rails_replica_sql"
      - type="patroni"
    target_matchers:
      - component="puma"
      - type="api"
  - equal:
      - env
      - environment
      - pager
    source_matchers:
      - component="rails_primary_sql"
      - type="patroni"
    target_matchers:
      - component="rails_requests"
      - type="api"
  - equal:
      - env
      - environment
      - pager
    source_matchers:
      - component="rails_replica_sql"
      - type="patroni"
    target_matchers:
      - component="rails_requests"
      - type="api"
  - equal:
      - env
      - environment
      - pager
    source_matchers:
      - component="rails_primary_sql"
      - type="patroni"
    target_matchers:
      - component="workhorse"
      - type="api"
  - equal:
      - env
      - environment
      - pager
    source_matchers:
      - component="rails_replica_sql"
      - type="patroni"
    target_matchers:
      - component="workhorse"
      - type="api"

Affected alerts

Reference: https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/15767

Edited by Steve Xuereb

Merge request reports