2025-10-10: Workhorse web requests apdex below SLO target

Workhorse web requests apdex below SLO target (Severity 3 (Medium))

Problem: The workhorse component for web requests had an apdex score below its SLO due to excessive broadcast messages, leading to web performance degradation.

Impact: The apdex score for the web workhorse component dropped below target, with 97.87% for the canary stage and 98.54% for the main stage over the last 6 hours. Users experienced slower web page loads and periods of reduced satisfaction.

Causes: A recent code change caused broadcast messages to be created per project or group when webhooks were rate-limited, resulting in an excessive number of broadcast messages and increased web request latency.

Response strategy: We reverted the problematic code change and are manually cleaning up excess broadcast messages from the database. Both workhorse web apdex alerts have cleared, and dashboards show improving apdex scores. Manual cleanup steps and backup procedures have been documented for ongoing mitigation until the revert is fully deployed.

---

This ticket was created to track INC-4684, by incident.io 🔥