Skip WebHooks::LogExecutionService lease for healthy hooks
What does this MR do and why?
WebHooks::LogExecutionService#update_hook_failure_state takes a
per-hook Gitlab::ExclusiveLease for every delivery. For a successful
delivery to a hook with no recent failures and no disabled_until,
enable! writes nothing, so the lease protects no write. Under load,
jobs wait for up to 25 retries of 0.25 seconds for this lease.
Behind the skip_web_hook_failure_state_lock_when_healthy flag, skip
the lease and the reload in that case. The service still calls
update_last_webhook_failure, so a stale failure banner still clears.
Failed deliveries, and successful deliveries to hooks with failures,
still use the lease.
The fast path decides from the hook as the worker loaded it.
WebHooks::LogExecutionWorker uses data_consistency :delayed, so
that row can come from a replica that does not have a recent failure
yet. In that case, or when a concurrent failure increments the counter
while a successful delivery skips the lease, the counter stays
incremented. The lease also allows this result (success first, then
failure), so the state stays correct. The next success resets it.
References
Screenshots or screen recordings
| Before | After |
|---|---|
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.