2025-09-09: rails_request error rate violating SLO on ai-assisted main stage

rails_request error rate violating SLO on ai-assisted main stage (Severity 2 (High))

Problem: The ai-assisted service in the main stage experienced elevated error rates due to decryption failures, causing key AI features to be unavailable.

Impact: A subset of users of AI-assisted features, including code completions, code generation, and chat functionality like Duo Chat and workflow in the browser, were unable to access code suggestions or generate responses. Errors were observed across the code_completions and agentic chat endpoints. After rollback, we are no longer seeing performance degradation for code suggestion functionality.

Causes: A recent deployment set Rails 7.1 as the default configuration, which led to a decryption failure of the cloud connector key.

Response strategy: We completed a rollback of the deployment, which immediately resolved the errors. Status page updates were communicated. More information can be found in the incident issue: #20499 (closed)

---

This ticket was created to track INC-3802, by incident.io 🔥