Unified load management in Gitaly

Olivier Campeau
2026-04-04T01:08:29Z

## Summary

When Gitaly experiences periods of extreme resource pressure — whether from traffic spikes, runaway git operations, or cascading failures — it can protect itself by rate limiting upcoming requests. This is done using the adaptative limiter or/and the concurrency limiter.

However, once admitted (in-flight) requests are pressuring the system, Gitaly currently lacks a mechanism to protect itself by actively shedding load.

This epic tracks the work for a **unified load management system** in Gitaly.

### Problem

Gitaly runs as a critical storage layer for GitLab. Under heavy load, git operations can accumulate — consuming CPU cycles and memory faster than they can be released — pushing the node into an unrecoverable degraded state. Without an automated pressure-relief mechanism, operators are left with manual intervention as the only recourse, often too slow to prevent broader impact.

Specifically:

* In-flight RPCs and their spawned git processes continue running even as cgroup memory and CPU limits are being breached.
* There is no existing mechanism to track which RPCs are contributing the most load and prioritize which to kill.
* Without active shedding, Gitaly may become unresponsive, causing widespread repository access failures across GitLab.

### Proposed solution

Implement a new **Unified load management system** within Gitaly. This system introduces a couple of new components, while adapting others.

#### Load Monitor

Currently, resource usage monitoring within Gitaly is performed by the cgroup manager. Our current rate limiters (adaptative limiter and concurrency limiter) uses the cgroup manager to detect pressure.

A new **LoadMonitor** will be introduced, that will centralize resource usage monitoring within Gitaly. The resource usage monitoring responsibility of the cgroup manager will be migrated to this LoadMonitor.

The responsibility of the LoadMonitor will be to monitor resource usage and emit events when certain conditions are met. It will be made extensible so that future resource usage monitoring needs could also be handled by it.

This design will allow to have a single component responsible for all resource usage monitoring within Gitaly, and generic enough to be used by any other components within Gitaly.

Our current rate limiters will be upgraded to use this new LoadMonitor instead of the current Watchers.

#### Load Shedder

The unified load management system also introduces a **Load Shedding** system for Gitaly. Using the LoadMonitor, it will configure thresholds that when reached, will trigger terminating in-flight RPCs and their associated git subprocess trees to restore the system to a stable, operable state.

### Design and architecture

See the following MR with the proposed design and architecture: https://gitlab.com/gitlab-org/gitaly/-/merge_requests/8568