Skip WebHooks::LogExecutionService lease for healthy hooks

What does this MR do and why?

WebHooks::LogExecutionService#update_hook_failure_state takes a per-hook Gitlab::ExclusiveLease for every delivery. For a successful delivery to a hook with no recent failures and no disabled_until, enable! writes nothing, so the lease protects no write. Under load, jobs wait for up to 25 retries of 0.25 seconds for this lease.

Behind the skip_web_hook_failure_state_lock_when_healthy flag, skip the lease and the reload in that case. The service still calls update_last_webhook_failure, so a stale failure banner still clears. Failed deliveries, and successful deliveries to hooks with failures, still use the lease.

The fast path decides from the hook as the worker loaded it. WebHooks::LogExecutionWorker uses data_consistency :delayed, so that row can come from a replica that does not have a recent failure yet. In that case, or when a concurrent failure increments the counter while a successful delivery skips the lease, the counter stays incremented. The lease also allows this result (success first, then failure), so the state stays correct. The next success resets it.

References

Screenshots or screen recordings

Before After

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Phil Hughes

Merge request reports

Loading
Loading