2026-06-30: server_code_completions error rate 31.87% in europe-west2

server_code_completions error rate 31.87% in europe-west2 (Severity 2)

Problem: The ai-gateway server_code_completions SLI in europe-west2 was returning 5xx errors at a rate of 31.87%, breaching error SLOs and causing failures for code completions.

Impact: Users in europe-west2 were unable to use code completions in Code Suggestions, with 31.87% of requests failing for about an hour. The problem affected multiple namespaces and was not limited to a single customer. Some users saw additional non-working model options during the incident.

Causes: A recent code change upgrading the tree-sitter/grammar library in the ai-gateway service introduced an I/O or permission error in the post-processing path. This resulted in elevated 500 errors for code completions. The issue was confirmed as self-inflicted and resolved by reverting the problematic change.

Response strategy: We deployed a revert of the recent grammar-related code change. The fix was successfully tested in staging and then rolled out to production, with error rates dropping as traffic shifted to the new deployment. A workaround was available for users to switch to the Claude Sonnet 4.5 model until the fix was in place. User communications and instructions were provided as needed.

This ticket was created to track INC-11636, by incident.io 🔥