Specify SLO target by shard for Sidekiq
In https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/24341#note_1532892894, we had multiple alerts from sidekiq_execution
SLI for urgent-authorized-projects
where a worker's execution apdex repeatedly dropping. Since the apdex performance couldn't seem to meet the default 99.5% SLO, there was a request to lower the threshold for that particular shard.
Without the capability to specify SLO by shard, we have been temporarily silencing alerts for some shards. This can potentially deafen us from apdex drops in real incidents. Thus, this is also a corrective action for production#17030 (comment 1619087522).
Edited by Gregorius Marco