Skip to content

fix(alerts): ignore shards in sidekiq traffic cessation

Steve Xuereb - Out of Office Back 2024-06-16 requested to merge fix/traffic-absent-shard into master

What

Ignore the shards below for the trafficCessation alerts:

  • gitaly-throttled
  • database-throttled
  • quarantine
  • urgent-authorized-projects
  • elasticsearch
  • memory-bound

Why

In gitlab-com/gl-infra/production#16956 (closed) we see gitaly-throttled cessation alert firing because we stopped running a background migration of the Gitaly servers. There was nothing wrong with the queue itself.

As we see in !6373 (merged) we used to be able to define trafficCessationAlertConfig per shard, and some of them had it disabled, but in !5654 (merged) we changed how we define the SLI for the shard and didn't update the traffic cessation alert.

Reference: https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/24555

Merge request reports