2026-10-15: Blackbox probe availability for customers.gitlab.com/-/liveness/database,migrations,cache below 70%

Blackbox probe availability for customers.gitlab.com/-/liveness/database,migrations,cache below 70% (Severity 2)

Problem: Blackbox probe availability for customers.gitlab.com /-/liveness/database,migrations,cache has fallen below 70% because CustomersDot became unresponsive.

Impact: Customers.gitlab.com /-/liveness/database,migrations,cache probes fell below 70%, and Customers Portal pages were intermittently unavailable or timing out for users.

Causes: We found that accumulated Prometheus metric store .bin files were making /metrics scrapes extremely slow, which saturated Puma CPU and threads and left CustomersDot unresponsive.

Response strategy: We removed accumulated Prometheus metric store .bin files on at least one VM and restarted Puma, which restored service there. We merged a revert to restore metric-file cleanup and started a CustomersDot deploy to apply the fix across the fleet.