Include patroni SLIs into the error budget for stage groups

Coming from https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6334#note_841860750

Currently, all SLIs that are sourcing the error budgets for stage groups are unweighted. This means that a Sidekiq job has the same value as a request. For now this has worked out.

But as we intend to include more components into the error budget for stage groups, and perhaps an event based availability number for GitLab.com (#1322 (closed)), then this won't work any more. For example if we want to include database queries (slow queries and errors) into the error budget, this would outweigh all other components in the availability number: we perform way more database queries than we do requests.

Suggestion:

Allow specifying a weight: field on a service level indicator definition. The weight defaults to 1. But can be adjusted to something between 0-1. We record this weight in a recording rule gitlab:component:weight that we can use in these availability calculations.

In the availability calculations we multiply the operation- and success rates with the associated weight. Causing a high traffic operation-rate not to outweigh others with a bigger impact.

Then we can include patroni clientside SLIs (rails_*_sql) labelled with a feature category into the error budget by setting feature_category: featureCategoryFromSourceMetrics.

Edited by Bob Van Landuyt