Add Reenqueuer concern
What does this MR do?
This is preparatory work for !20002 (merged).
It makes it easy to add new Workers that do unlimited work without running long, see #43664 (closed)
The shared examples can be easily added to any Worker that includes this concern.
Copied from the class comment:
A concern that helps run exactly one instance of a worker, over and over, until it returns false or raises.
To ensure the worker is always up, you can schedule it every minute with sidekiq-cron. Excess jobs will immediately exit due to an exclusive lease.
The worker must define:
#perform
#lease_timeout
The worker spec should include
it_behaves_like 'reenqueuer'
.Optionally override
#minimum_duration
to adjust the rate limit.When
#perform
returns false, the job will not be reenqueued. Instead, we will wait for the next one scheduled by sidekiq-cron.#lease_timeout should be longer than the longest possible
#perform
. The lease is normally released in an ensure block, but it is possible to orphan the lease by killing Sidekiq, so it should also be as short as possible. Consider that long-running jobs are generally not recommended. Ideally, every job finishes within 25 seconds because that is the default wait time for graceful termination.Timing: It runs as often as Sidekiq allows. We use ReenqueuerSleeper to sleep instead of scheduling later with
#perform_in
to enforce a rate limit, because sidekiq-cron would otherwise sometimes start a job prematurely.
Does this MR meet the acceptance criteria?
Conformity
- [-] Changelog entry
- [-] Documentation (if required)
-
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. - [-] Tested in all supported browsers
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
- [-] Label as security and @ mention
@gitlab-com/gl-security/appsec
- [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
- [-] Security reports checked/validated by a reviewer from the AppSec team