Skip to content

Increase lease acquisition wait time

Dominic Bauer requested to merge scan-result-policy-lock-acquisition-exp-backoff into master

What does this MR do and why?

We added lease acquisition to a worker in !110020 (merged) to guard from a race condition.

By default, Gitlab::ExclusiveLeaseHelpers#in_lock attempts to acquire the lease 10 times, with a backoff of 100ms each time.

Our worker has a median runtime of ~2.7s and 90% percentile is ~5.2s (source: Kibana, can't link to aggregations). As a result, we're seeing FailedToObtainLockError exceptions raised, because the lease is not acquired in time.

This MR increases the total backoff time for lease acquisition from 1s (10 * 0.1s) to 5s, using exponential backoff.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Dominic Bauer

Merge request reports