Skip to content

Three processors causing 4.5-minute webhook processing delays

Problem Statement

Webhook processing is taking 4.5+ minutes instead of seconds due to three processors spending 80-90+ seconds each in their applicable? methods, even when they don't match:

  1. Triage::EngineeringAllocationLabelsReminder - 84.56 seconds
  2. Triage::RequireTypeOnRefinement - 92.51 seconds
  3. Triage::ScheduledIssueTypeLabelNudger - 94.33 seconds

These three processors consume 271.4 seconds of the total 272.99 seconds (99.4%) while producing no results.

Impact: During traffic spikes, slow processors block all worker threads, causing up to 30-45 minute delays and cascade failures.

Goals

  1. Reduce applicable? execution time for these processors from 80-90+ seconds to under 1 second each
  2. Restore real-time processing - return webhook processing to under 10 seconds for typical events
  3. Maintain existing functionality - ensure processors work correctly for matching events

Next Steps

  1. Examine applicable? methods in these processors for expensive operations (API calls, database queries, etc.)
  2. Optimize by adding caching, early returns, or moving logic to process method
  3. Add timeouts to prevent runaway processes
Edited by David Dieulivol