Use soft SLO for generating warnings in capacity planning
We currently use the hard SLO for creating capacity planning issues. The hard SLO is also used for alerting the SRE oncall when this is breached.
Ideally, the soft SLO would warn us before we start alerting the on-call through in capacity planning. We've gotten a start on that in https://gitlab.com/gitlab-com/gl-infra/tamland/-/merge_requests/193. However, this would cause a lot of duplicate notifications, so we decided to block this on &821 (closed) so we can more reliably avoid duplicate issues.
We'll also need to go through all of the soft-SLO targets, they are likely to have become incorrect, as we've never used them before.