2025-10-28: Teleport connections failing with "Error while dialing: failed to dial: dial tcp 10.253.8.176:3025: i/o timeout"

Teleport connections failing with "Error while dialing: failed to dial: dial tcp 10.253.8.176:3025: i/o timeout" (Severity 4 (Low))

Problem: Teleport connections are failing with timeout errors.

Impact: Teleport requests and deployments, are failing with timeout errors. This is leading to intermittent connection failures for users and deployment jobs.

Causes: We are hitting a Google Cloud Storage rate limit for object mutation operations, which is causing the Teleport pods to occasionally crash.

Response strategy: The pods restarted on their own. We will continue to investigate why we're hitting the GCS rate limits.

---

This ticket was created to track INC-5307, by incident.io 🔥