2026-02-03: GraphQL query apdex SLO violation in cny stage

GraphQL query apdex SLO violation in cny stage (Severity 4)

Problem: GraphQL query performance in the cny stage of the API dropped below the defined service level objective, triggering multiple SLO alerts and cyclical performance degradation.

Impact: The apdex for known GraphQL query operations in the cny stage of the API fell below the service level objective between 06:00 and 14:30, resulting in degraded performance for users during this period. Cyclical drops in apdex continued, with multiple dips approaching SLO burn thresholds and full recovery not yet confirmed.

Causes: Long-running GraphQL requests in the 'continuous_integration', 'code_review_workflow', and 'pipeline_composition' feature categories contributed significantly to the Apdex drop. There were recurring dips in performance throughout the day. No single slow transaction, user, or anomalous traffic was identified.

Response strategy: A change was made to align API-CNY's CPU scaling with API-MAIN to allow more aggressive scaling in response to load. The system is being monitored for further cyclical drops and signs of recovery.

This ticket was created to track INC-7062, by incident.io 🔥