Make Gitaly per-node alerts less noisy
Currently the per-node Gitaly alerts use the same alerting SLO for per-node alerts as we do for Global gitaly alerts. But because the traffic can widely vary between different nodes, this threshold is too tight, causing too many alerts.
Proposal 1: Rely only on the Global SLO
We could look at historical data for a past incident that was an incident that needed attention, and see where we should put the global monitoring thresholds to make sure an alert would fire.
This would mean we can set a tighter global SLO and remove the per-node alerts. That means that the error budget for the Gitaly group becomes the same number that SREs are alerted on.
Proposal 2: Lower per-node SLOs
If we can't realistically tighten the Global SLO to discover these per-node events. We could also lower the per-node SLO to something scaled from the global SLO based on the number of nodes.