feat: add the metropolis metrics operator and the rig's SLIs

What does this merge request do and why?

Adds an opt-in fragment for the Metropolis GitLab metrics operator, and enough real configuration to exercise it: GitLabServiceMetrics resources for the four GitLab services this rig runs, ported from the runbooks GET-hybrid catalog with every metric name, label and threshold checked against the live Prometheus here.

The point is somewhere to develop Metropolis against a real GitLab rather than against the synthetic samples in its own repo. config-manager is switched off, since the operator now generates what the runbooks catalog would have.

Two things had to happen first. Nothing was setting type on any scraped series, and the catalog scopes every rule, alert and panel on it, so each monitor now stamps its own. Dashboards are delivered as GrafanaDashboard resources rather than sidecar ConfigMaps, which lifts the 1MiB cap a ConfigMap puts on a dashboard; that needs grafana-operator, so the fragment brings it along with an external-mode Grafana wrapping the one already deployed. Nothing runs a second Grafana.

The chart is at 2.3.1. Two things in the 2.x line shape this config: the chart can no longer install its own CRDs, so a metropolis-crds deployer applies them from the tag ahead of the release; and the values API was renamed wholesale in 1.29.0 and rejects the old names outright.

References

• Metropolis: https://gitlab.com/gitlab-com/gl-infra/sandbox/metropolis
• Broken Jsonnet renderer, filed from this work and fixed in 1.21.2: gitlab-com/gl-infra/sandbox/metropolis#62 (closed)

How to set up and validate locally

These steps also live in docs/fragments/metropolis.md now, so they outlast the MR.

Both charts involved are in private registries, so the two helm registry logins have to happen before caproni up or it fails pulling them.

1. git fetch && git checkout an/review-o11y-metrics-dashboards && mise install
2. helm registry login registry.gitlab.com — a PAT with read_registry, for the Metropolis chart and operator image.
3. helm registry login registry.ops.gitlab.net — for k8s-monitoring-stack. CI reads a shared read-only account from Vault for this; locally, use your own.
4. cp caproni.local.yaml.example caproni.local.yaml, then uncomment caproni.o11y-metrics.yaml and caproni.metropolis.yaml. Metropolis deploys standalone but only does anything useful next to o11y-metrics, which owns the Prometheus and Grafana it targets.
5. caproni up, then caproni update-etc-hosts and caproni ca trust.

Then, working down the deployer graph:

1. caproni kubectl get crd | grep metrics.gitlab.com — two definitions, applied by metropolis-crds before the release. This ordering is the whole point of that deployer: the chart renders a GitLabServiceMetrics for its own self-monitoring, and Helm resolves every kind against the API server before it creates anything, so the install fails outright if the CRDs are not already there.

2. caproni kubectl -n gitlab-metrics-system get pods,gsm — the operator Running, and the chart's own metropolis resource Ready. That resource is new in 2.x: the operator now monitors its own reconcile loop.

3. caproni kubectl -n gitlab get gsm — gitaly, redis, sidekiq and webservice, all Ready.

4. caproni kubectl -n gitlab get grafanadashboards — five, each with an empty NO MATCHING INSTANCES column, which is grafana-operator confirming it matched them to the external Grafana across the namespace boundary.

5. Open http://grafana.caproni.test (mise run grafana:adminpassword) and look in the gitlab folder: four service Overview dashboards plus "Caproni Rig Overview", the hand-written Jsonnet one.

6. A rule that selects nothing still reports health: ok, so query rather than read the rules page. Port-forward Prometheus, which has no ingress:

   caproni kubectl -n monitoring port-forward svc/prometheus-operated 9090:9090
   curl -s localhost:9090/api/v1/query \
     --data-urlencode 'query=count by (type) ({__name__=~"gitlab_component_.*"})' | jq '.data.result'

   One row per service. The service-level rollups (gitlab_service_ops:rate_5m) need an hour of data before they report anything, so an empty headline row on a fresh rig is expected rather than broken.

7. The two knobs taken from 2.1.0 and 2.2.0:

   caproni kubectl -n gitlab get gsm webservice \
     -o jsonpath='{.spec.serviceLevelIndicators.graphql_query.serviceAggregation}'   # false
   caproni kubectl -n gitlab-metrics-system get cm metropolis-config \
     -o jsonpath='{.data.config\.yaml}'                                              # carries alertTuning

   The first stops GraphQL requests being counted twice in the webservice headline row, since puma already counts them. The second lowers the traffic floor an SLI must clear before its SLO alerts may fire; at the vendored default of 3600 samples over the long window, nothing on a rig this quiet ever clears it and the alerts are silent while their rules report healthy.

Tearing down is caproni destroy. The CRDs are cluster-scoped and deleting them garbage-collects every GitLabServiceMetrics with them, which on this rig is only the resources in manifests/metropolis-gitlab.

Screenshots

Gitaly overview custom dashboard http://grafana.caproni.test/d/gitaly-overview/gitaly-overview?orgId=1&from=now-6h/m&to=now/m&timezone=utc&var-PROMETHEUS_DS=default

Caproni Rig overview custom dashboard http://grafana.caproni.test/d/caproni-rig-overview/caproni-rig-overview?orgId=1&from=now-6h&to=now&timezone=utc&var-PROMETHEUS_DS=prometheus

<🤖>

Validated on the rig: 989 rules loaded with 0 unhealthy, and all four services produce gitlab_component_* series.

The chart moved 1.20.0 to 2.3.1 during review. Six releases changed what this rig should say:

1. 1.21.2 fixed the Jsonnet renderer (work item 62 above), so the hand-written overview dashboard is no longer held back.
2. 1.22.0 added selector.expressions, the metrics-catalog operand hash. Several SLIs here were knowingly coarser than the catalog because a selector was exact-match only, and each said so in a comment. Those approximations are gone: gitaly's error rate is upstream's ten-code noneOf rather than a single code, workhorse's is code =~ '^5.*' rather than code="500", and graphql excludes endpoint_id = 'graphql:unknown'.
3. 1.23.0 generates saturation alerts. That turns a point which can never produce a ratio from something inert into an alert that can never fire, so the saturation lists are cut to the eight points that actually resolve here. The declared set and the set Prometheus reports are now identical.
4. 1.29.0 renamed the values API and rejects the old names. One of the eight renames is used here, metrics.serviceMonitor.labels to additionalLabels. The rig's resources.limits.memory override went in the same change: 512MiB is now the chart default, which is the fix for the OOM loop this branch hit at the old 128MiB.
5. 2.0.0 dropped the chart's ability to install the CRDs, hence the metropolis-crds deployer and the second version pin that Renovate keeps in step with the first.
6. 2.1.0 and 2.2.0 added serviceAggregation and globalConfig.alertTuning, both described in the steps above. 2.3.0 also changed what links.runbookBaseUrl commits you to — setting it now requires runbookPath on every SLI — so it is unset, and runbooks resolve to runbooks.gitlab.com by default.

Prometheus-monitoring-itself was dropped from the suite. It earned its place while the SLIs were being validated, being the only service whose rules certainly aggregated real series, but it describes the rig's own tooling rather than GitLab.

Grafana's database is now on a PVC. It was an emptyDir, and the Grafana pod restarts on every upgrade of k8s-monitoring-stack (the chart's checksum/secret annotation changes each time), so each caproni up left Grafana with no dashboards while get grafanadashboards still reported them applied. grafana-operator only re-pushes to an external instance on its 10m resync.

</🤖>