## Problem Instance-wide Geo Prometheus metrics are cumbersome and inaccurate, because metrics are inherently tied to hosts. ### Background `Geo::MetricsUpdateWorker` is a Sidekiq job enqueued by `sidekiq-cron` once per minute on Geo sites. It takes an exclusive lease so only one instance runs at a time. It: - collects many metrics (e.g. `geo_project_repositories_synced`) - if in a secondary site, it reports them to the primary site via `GeoNodeStatus` - exports them as Prometheus gauges on the Sidekiq exporter's `/metrics` endpoint The problem is that in a large environment, there are many Sidekiq processes across different hosts. Each process exports the same Prometheus metrics, but only the process that last ran the worker has fresh values. The others serve stale data. Since Prometheus ties each data point to its `{instance}` label, this results in multiple fragmented time series for what should be a single metric. To get a single line in Grafana, we are forced to use aggregations like `MIN()` or `MAX()` across instances, **but these are not totally truthful**. The lease holder changes every minute as different Sidekiq processes pick up the job, so there is no stable instance to filter on. Prometheus has no native concept of "connect the latest dot regardless of which instance emitted it." And it's a pain to work with these metrics e.g. in Grafana. This is a significant challenge to work around and is at least one direct cause of **inaccurate alerting, especially when it comes to replication lag**. In Disaster Recovery, RPO is sensitive to replication lag. This is particularly impactful to GitLab Dedicated. ## Proposal **Export Geo metrics from a single host: [`gitlab_exporter`](https://gitlab.com/gitlab-org/ruby/gems/gitlab-exporter)** The data already exists: `Geo::MetricsUpdateWorker` reports metrics to the primary Geo site, which stores them in the `geo_node_statuses` table. `gitlab_exporter` runs as a single process (a single pod in Kubernetes environments, or a single service in Linux Package installations) that is only accessible internally and is already scraped by Prometheus. By adding a new database probe that reads `geo_node_statuses` and emits Prometheus gauges labeled by Geo site (e.g. `site` or `geo_node`), we get: * **One continuous time series per site per metric** since `gitlab_exporter` is a single stable exporter with a consistent `{instance}` label. * **No new infrastructure** such as Pushgateway or custom endpoints. * **Truthful data** since each scrape reads the latest status reported by the Geo site directly from Postgres. * **Built-in staleness detection** via the existing `last_successful_status_check_at` timestamp in `geo_node_statuses`. Note: the original proposal suggested using the primary's web server `/-/metrics` endpoint, but as @cmiskell [pointed out](https://gitlab.com/gitlab-org/gitlab/-/work_items/590761#note_3099587103), that has the same fragmentation problem since we scrape each individual webservice pod, not a single front-door endpoint. `gitlab_exporter` avoids this because it runs as a single process. The existing Sidekiq-exported Geo gauges should be kept indefinitely. Not all environments run `gitlab_exporter` (it is optional), and anyone currently depending on the Sidekiq metrics would need to intervene to switch. Deprecating them would require a long-term plan, is unnecessary, and is out of scope here. The goal is to provide a **better alternative** for environments that do run `gitlab_exporter`, not to remove the existing path. ### Implementation Plan *Mike 2026-02-19: The first two items of this plan were written by AI. We need to confirm their accuracy when refining this issue and before adding a weight.* - [x] **Add a `Database::GeoNodeStatusProber`** to [gitlab-exporter](https://gitlab.com/gitlab-org/ruby/gems/gitlab-exporter). Following the existing probe pattern (similar to `RemoteMirrorsProber`): - A `GeoNodeStatusCollector` that queries `SELECT * FROM geo_node_statuses` (or a targeted column list). - A `GeoNodeStatusProber` that emits each numeric column as a Prometheus gauge, labeled by `geo_node_id` / site name. - The metric names can be derived from the column names, which are already kept in sync with the `GeoNodeStatus::PROMETHEUS_METRICS` constant by the Rails app. This avoids maintaining a separate metric list in `gitlab_exporter`. - [ ] **Configure the probe** in the `gitlab_exporter` YAML config (e.g. under the `database` endpoint or a new `geo` endpoint), passing the database connection string. - [ ] **Announce the metrics** (in release post?) stating the problem they solve and to therefore update Prometheus scrape targets / Grafana dashboards for environments that run `gitlab_exporter` to consume the new single-source metrics instead of the fragmented Sidekiq exporter metrics.