2026-01-07: Manifest read requests on registry manifest endpoints in us-east1-b region below apdex SLO

Manifest read requests on registry manifest endpoints in us-east1-b region below apdex SLO (Severity 4)

Problem: Read requests to the manifest endpoints for the registry are not meeting apdex SLO targets due to some requests experiencing high latency.

Impact: A small percentage of users experience slower response times when fetching image manifests. This appears to be a recurring pattern starting around 2025-12-13.

Causes: The main cause is a memory leak in the Redis SDK used by Registry. As each Registry pod ages, this slow leak gradually bloats memory usage to the point where memory pressure and reclaim cause request processing to slow down. Upgrading the SDK fixes the bug. Separately, a much smaller contributing cause may be a specific slow SQL query that incurs significant disk IO under certain conditions.

Response strategy: Upgrading Registry's version of the Redis SDK fixes the memory leak and resolves the latency it eventually causes. See gitlab-org/container-registry#1799 (closed) for details. This fix has been deployed to canary and shows good results; promoting it to main stage production should resolve this incident.

This ticket was created to track INC-6480, by incident.io 🔥