Add runway-db-smgke-dr service, dashboard and runbook

What this does

Registers the Cloud SQL PITR clone runway-db-smgke-dr (in gitlab-runway-staging) in the metrics catalog and the service catalog, so it gets a service overview dashboard and the SLI recording rules that dashboard reads.

Why

The clone exists to exercise the Cloud SQL disaster-recovery path for secrets-manager-gke: clone the primary, repoint the workload at the clone, confirm OpenBao unseals, then test the last-resort in-place restore. Tracked in dbo-issue-tracker#771, with the maintenance-window plan in dbo-issue-tracker#792.

Cloud SQL telemetry is labelled with type set to the instance name, so the raw metrics for the clone already flow — for example:

sum by (transaction_type) (rate(stackdriver_cloudsql_database_cloudsql_googleapis_com_database_postgresql_transaction_count{type="runway-db-smgke-dr", database="openbao", env="gstg"}[5m]))

But the Grafana dashboard and the gitlab_service_ops:* / gitlab_component_ops:* recording rules the dashboard actually queries are generated from this repository and keyed on the instance name. Today gitlab_component_ops:rate_5m{type="runway-db-smgke-dr"} returns nothing, so there is no dashboard and no SLI.

There is a second reason to do this properly rather than skip it. If a real DR event ever promotes a clone to be the live instance, the instance name changes, and exactly these five files have to change with it. None of them can be created by Terraform or Runway. This MR is the rehearsal for that, so the procedure is proven before it is needed.

What changed

Six changes. Every hand-written file is a copy of the equivalent file for the primary instance runway-db-secrets-manager-gke with only names changed, so each one can be diffed line by line against its source.

File	New or edit	Copied from	What changed
metrics-catalog/services/runway-db-smgke-dr.jsonnet	New file	metrics-catalog/services/runway-db-secrets-manager-gke.jsonnet	local type → runway-db-smgke-dr; descriptiveName gained (DR clone); nothing else. SLI, thresholds, selector and comments untouched
metrics-catalog/services/all.jsonnet	Edit	—	One line added, import 'runway-db-smgke-dr.jsonnet',, directly after the primary's import so the list stays alphabetical
dashboards/runway-db-smgke-dr/main.dashboard.jsonnet	New file, new folder	dashboards/runway-db-secrets-manager-gke/main.dashboard.jsonnet	The name inside serviceDashboard.overview(); nothing else. The file is six lines
services/service-catalog.yml	Edit	—	Copied the 18-line - name: runway-db-secrets-manager-gke block and pasted it directly below. Changed five values in the copy: name, friendly_name, label (RunwayDBSMGKEDR, must be unique in the catalog), the instance name inside the logging permalink, primary_grafana_dashboard. The original block is untouched
docs/runway-db-smgke-dr/README.md	New file, new folder	docs/runway-db-secrets-manager-gke/README.md	Verbatim copy, same 90 lines. Instance name throughout, the # heading to match the new friendly_name, and the label. Substituting the names back produces a file identical to the source. It therefore inherits the primary's statements about backup schedule, restore validation and alert routing
mimir-rules/runway/runway-db-smgke-dr/ — 13 files	New, generated	Not copied — output of make generate	Generated from the metrics-catalog entry above, so nothing here is hand-written. File list and commands below

The 13 generated files, and how to regenerate them

Run this after the service definition exists, because the rules are generated from it:

mise trust && mise install     # pinned toolchain from .tool-versions; needs coreutils for nproc on macOS
mise exec -- make generate     # runs scripts/bundler.sh (jb install), then scripts/generate.sh
git status --short             # expect one new directory with 13 files
git add mimir-rules/runway/runway-db-smgke-dr/

make generate deletes and rebuilds every *autogenerated*.yml under mimir-rules/, then runs four generators in parallel: generate-jsonnet-rules.sh, generate-docs, generate-all-reference-architecture-configs.sh and generate-service-dashboards. Running it locally can also remove files under reference-architectures/get-hybrid/config/; that is local-environment churn unrelated to this change, so revert it with git checkout -- reference-architectures/ before committing.

CI enforces the result with ensure-generated-content-up-to-date, which re-runs make generate and fails with Please run 'make generate' on any diff or untracked file.

Files:

autogenerated-runway-runway-db-smgke-dr-component-aggregation.yml
autogenerated-runway-runway-db-smgke-dr-feature_category-aggregation.yml
autogenerated-runway-runway-db-smgke-dr-occurrence-sla-availability.yml
autogenerated-runway-runway-db-smgke-dr-runway-db-smgke-dr-service-slos.yml
autogenerated-runway-runway-db-smgke-dr-saturation-alerts.yml
autogenerated-runway-runway-db-smgke-dr-saturation-metadata.yml
autogenerated-runway-runway-db-smgke-dr-saturation.yml
autogenerated-runway-runway-db-smgke-dr-service-anomaly-detection-alerts.yml
autogenerated-runway-runway-db-smgke-dr-service-level-alerts.yml
autogenerated-runway-runway-db-smgke-dr-service_anomaly_detection.yml
autogenerated-runway-runway-db-smgke-dr-sli-aggregations.yml
autogenerated-runway-runway-db-smgke-dr-transformed-service-aggregation.yml
autogenerated-runway-runway-db-smgke-dr-transformed-service_component_stage_groups-aggregation.yml

Verification

• generate-dashboards-cache and test-dashboards must pass — the latter runs dashboards/upload.sh -D and make validate-service-dashboards.
• validate-service-catalog and validate-service-catalog-schema cover the catalog entry. Confirmed locally that the file still parses and that both the service name and the RunwayDBSMGKEDR label are unique across all 158 services.
• The rendered dashboard JSON is browsable from the prepare-service-health-dashboard-json artifact (service health dashboard manifest, under generated/) if you want to inspect the panels before merge.
• After merge, the dashboard appears at https://dashboards.gitlab.net/d/runway-db-smgke-dr-main/runway-db-smgke-dr3a-overview. Select gstg in the environment dropdown — the clone only exists in staging.

Notes for the reviewer

• The dashboard will be empty until the recording rules generate. deploy-dashboards is main-branch-only, so nothing reaches Grafana before merge, and the SLI rules need a cycle after that. This needs to be merged ahead of the maintenance window, not during it.

• Links point at gitlab-runway-production, deliberately. The Cloud SQL and Logs Explorer shortcut links follow the existing service. This clone currently exists only in gitlab-runway-staging, so those two links return nothing today. They are kept as-is because in a real DR scenario the instance exists in both staging and production, and this entry is meant to be identical to what a real cutover would add. Dashboard data is unaffected — select gstg in the environment dropdown and every panel renders.

• Merging creates a group label. reconcile-service-catalog-labels runs on master when services/service-catalog.yml changes and creates any missing Service::* label in the gitlab-com/gl-infra group, so Service::RunwayDBSMGKEDR will be created for what is a temporary instance. Say if you would rather that did not happen.

• This adds SLIs and error-budget tracking for a temporary instance. The SLI is severity: s4 and userImpacting: false, so it does not page. When the DR exercise finishes and the clone is torn down, these five entries should be removed in the same change that drops it from managed.yml.