2026-02-02: High error rate for anthropic model inferences in ai-gateway

High error rate for anthropic model inferences in ai-gateway (Severity 3)

Problem: Model inferences through the ai-gateway were failing at a high rate due to client errors.

Impact: Features relying on the /v1/proxy/anthropic/v1/messages endpoint, including external agents like Claude Code and OpenCode, could not complete model inferences for several hours. Codex models were not impacted.

Causes: A recent proxy endpoint refactor introduced an issue with HTTP client caching in LiteLLM. When the cached client expired, new requests encountered a 'client has been closed' error, leading to high error rates for affected proxy endpoints. The root cause was the use of a monkey-patched HTTP client, which interfered with LiteLLM's client cache and cleanup logic.

Response strategy: We reverted the recent proxy endpoint change and removed the monkey patching workaround. This restored normal functionality and the error rate returned to 0%.

This ticket was created to track INC-7017, by incident.io 🔥