2026-06-19: Gitaly active node available space saturation exceeding SLO
# Gitaly active node available space saturation exceeding SLO (Severity 3)
**Problem**: Available space on Gitaly active nodes in gprd/main/default exceeded the service SLO and remained near the hard threshold, risking capacity for new repositories.
**Impact**: Reduced headroom for placing new repositories on GitLab.com. There was a risk of running out of space for new projects around July 4–5, 2026 (UTC) if additional capacity was not added. The missed rebalancing may have further affected storage distribution, though no outages have occurred.
**Causes**: Gradual, ongoing capacity drift on /var/opt/gitlab across the default-shard active nodes resulted in persistent low headroom. Additionally, the shard rebalancer had not been running for about a month due to an invalid token, which impacted weight distribution and capacity forecasting.
**Response strategy**: We have provisioned and configured 9 new Gitaly storage nodes in gitlab-gitaly-gprd-2650 and restored the shard rebalancer by fixing its token. Chef runs were successful. We are now validating that the new nodes are eligible (above 20% free-disk threshold) and monitoring the saturation ratio. We will add an alert to notify if the shard rebalancer does not succeed for an extended period.
_This ticket was created to track_ [_INC-11283_](https://app.incident.io/gitlab/incidents/11283)_, by_ [_incident.io_](https://app.incident.io) 🔥
issue