Three processors causing 4.5-minute webhook processing delays
Problem Statement
Webhook processing is taking 4.5+ minutes instead of seconds due to three processors spending 80-90+ seconds each in their applicable? methods, even when they don't match:
-
Triage::EngineeringAllocationLabelsReminder- 84.56 seconds -
Triage::RequireTypeOnRefinement- 92.51 seconds -
Triage::ScheduledIssueTypeLabelNudger- 94.33 seconds
These three processors consume 271.4 seconds of the total 272.99 seconds (99.4%) while producing no results.
Impact: During traffic spikes, slow processors block all worker threads, causing up to 30-45 minute delays and cascade failures.
Goals
-
Reduce
applicable?execution time for these processors from 80-90+ seconds to under 1 second each - Restore real-time processing - return webhook processing to under 10 seconds for typical events
- Maintain existing functionality - ensure processors work correctly for matching events
Next Steps
- Examine
applicable?methods in these processors for expensive operations (API calls, database queries, etc.) - Optimize by adding caching, early returns, or moving logic to
processmethod - Add timeouts to prevent runaway processes
Edited by David Dieulivol