Improve resilience of monolithic Prometheus instances

GitLab Infrastructure Service - incident.io
2026-03-25T22:13:38Z

In the related incident our regional cluster Prometheus instances crashed due to hitting their memory limits.

While the direct cause of this was saturation, we need to improve how we deploy and shard Prometheus to:

- Reduce failure domain
- Improve scalability

## Short Term Solution
Right now the regional cluster Prometheus instances are doing a lot of work. They don't just scrape the in-cluster metrics, but are also responsible for our external scrape, for example - GCE VM discovery and collection. Because of this they are doing a lot more work than our other clusters.

As a short term solution we could move all the external scrape jobs to their own set of HA prometheus instances. However we would need to update the deduplication labels in use by Mimir.

Currently within Mimir we use the default deduplication labels of `cluster` and `__replica__`. Unfortunately we also rely heavily on the `cluster` label to denote our kubernetes cluster names. We should ideally update this to use `k8s_cluster_name` instead to follow [open telemetry semantic naming](https://opentelemetry.io/docs/specs/semconv/registry/attributes/k8s/), but this creates a bigger challenge around the relabeling of historical data.

Instead we can set the `ha_cluster_label` per tenant.

What this would look like:

- Add a new label to Prometheus external labels: `prometheus_cluster`.
- Update the tenant config for gitlab-gprd to use `ha_cluster_label: prometheus_replica`
- Create two new Prometheus instances with selectors that won't pick up any Prometheus jobs (CRDS)
- Create a MR to switch the selectors for `ScrapeConfigs` to the new instances, and disable in the old
- We would also move over any static scrape configs in the configuration with this.

We should also replicate the Mimir `ha_cluster_label` change across all our tenants. We can later set it to the new default once we validate no tenant is still using it.

### Limitations

While this helps reduce the load by splitting out some of the jobs, it does now help the in-cluster scalability or failure domain. For this we should use the long term solution below.

## Long Term Solution

For the long term solution we need to move to a daemonset based deployment. This drastically improves scalability and the failure domain of Prometheus, as scraping is now tied to the node. Each Prometheus instance now only scrapes and collectors metrics from node local pods. Since this uses a daemonset, it also scales as the cluster does (as new nodes are provisioned).

We still need to also implement, in part, the short term solution for external scrape jobs and cluster wide jobs. I.e scrape jobs that can not easily be run on a node local collector (daemonset).

This can optionally be done with Prometheus, or ideally by moving to the Open Telemetry Collector where we can leverage the target allocator.

What this would look like:

- Add a new label to Prometheus external labels: `prometheus_cluster`.
- Update the tenant config for gitlab-gprd to use `ha_cluster_label: prometheus_replica`
- Create a new Prometheus instance OR open telemetry deployment.
- Move all cluster wide / external jobs to the new deployment following the same method as the short term solution.
- NOTE: The open telemetry collector at this stage does not support ScapeConfig CRs with the targetAllocator (that might change).
- At this point the old Prometheus HA instances should only be responsible for ServiceMonitors and PodMonitor objects.
- Create a new daemonset based deployment and deploy this while removing the old one.

## Priority

This issue around scalability within our Prometheus monolithic instances is a long standing problem and has been raised several times. We have always opted to just keep increasing resources so the problem goes away. But it does not solve the issue and we really should be prioritizing this to permanently solve this problem.

_This ticket was created from_ [_INC-8732_](https://app.incident.io/gitlab/incidents/8732) _using_ [_incident.io_](https://app.incident.io) 🔥