2026-02-06: Elevated error rate on runway_ingress in europe-west3 region (12.21%)

Elevated error rate on runway_ingress in europe-west3 region (12.21%) (Severity 3)

Problem: A sustained elevated error rate is affecting runway_ingress in the europe-west3 region, with failures on the /v1/proxy/anthropic/v1/messages endpoint due to requests exceeding allowed token limits. Errors occur across multiple Anthropic models and persist despite previous mitigation steps.

Impact: Requests to the /v1/proxy/anthropic/v1/messages endpoint in the europe-west3 region continue to fail with 400 Bad Request errors from Anthropic, with the current error rate breaching the service SLO. Most errors are still concentrated in a single self-managed GitLab instance.

Causes: Requests to multiple Anthropic models, including Claude Opus and Sonnet, have been exceeding the 200k token limit set by the provider. An update to the token counting strategy may have contributed to incorrect estimation of token counts, resulting in oversized prompts and 400 Bad Request errors from Anthropic. The model rollback did not resolve the issue, and reverting the recent token counting change is in progress to confirm if it resolves the prompt length errors.

Response strategy: We rolled back the model selection change and confirmed the revert was deployed to production, but the error rate did not improve. A new merge request to revert the recent token counting strategy update has been created and is being deployed to determine if this resolves the prompt length errors.

We're resolving this issue as its no longer exceeding thresholds, and we'll open a followup for a more comprensive investigation regarding the newer errors.

This ticket was created to track INC-7171, by incident.io 🔥