Incorporate gitaly "active shards" into capacity planning
We have an alert for "number of active gitaly shards" which triggered during my shift: production#7077 (closed).
The gitaly-shard-weights-assigner checks the gitaly fleet for disk space. Any node that is above 80% disk space utilization drops out and no longer gets any new repos assigned (the 20% is buffer for growth on the existing repos, we get a separate alert once we reach 90%, at which point we trigger a rebalance).
The alert in question checks if we have enough gitaly shards that are still receiving new projects. Effectively it looks at how many nodes have disk space <80%. And if the count of those nodes is <10, it triggers the alert.
This is effectively a capacity planning mechanism. We should figure out if we can incorporate this into our capacity planning / saturation framework. Instead of paging the on-call, we should address these proactively.