Check `needs_processing?` inside lease of AtomicProcessingService

Leaminn Marequested to merge
atomic-processing-obtain-lease-before-query into master

What does this MR do and why?

Context

The needs_processing? query in AtomicProcessingService is being executed at a very high rate. We seek to reduce this call rate to avoid potential LWlock contention (#598584). This MR addresses one of the proposed changes.

Currently, in AtomicProcessingService, we are checking needs_processing? before obtaining a lease. Upon lease acquisition failure, the service simply exits without first processing. This means that the needs_processing? query is needlessly being executed on this code path.

This MR

In this MR, we update AtomicProcessingService so that we only check needs_processing? inside the lease. We estimate ~16.6% reduction in needs_processing? calls. This is the first iteration towards reducing this query's call rate.

This change is made behind a feature flag: ci_atomic_processing_check_inside_lease. Roll-out issue: #599796

References

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #598584

Edited by Leaminn Ma

Merge request reports