Add hooks for deferred and resumed concurrency limit jobs

What does this MR do and why?

Adds two optional class methods that a worker with a concurrency limit can define:

  • on_concurrency_limit_deferred(args): called when a job is deferred.
  • on_concurrency_limit_resumed(args): called when a deferred job is resumed.

A job can depend on state that expires while it waits in the concurrency limit queue. These hooks let the worker keep that state alive. The first planned user is CreatePipelineWorker, which needs to keep its pipeline creation request alive while it waits under a per-user limit.

Details
  • The hooks receive the job arguments, uncompressed when the size limiter compressed them. The stored job stays compressed.
  • Errors in a hook are tracked with Gitlab::ErrorTracking and not raised, so a hook cannot lose a deferred job or stop a resume.
  • The hooks work with any concurrency limit, with or without a scope.
  • deferred is called from Middleware#defer_job!. resumed is called from QueueManager#prepare_and_store_metadata, before the job is pushed back to Sidekiq.

References

Problem: https://gitlab.com/gitlab-org/gitlab/-/work_items/604640

Step Merge request What it does Needs
1 !264227 (merged) (merged) Adds concurrency_limit_scope, a per-namespace or per-user share of a worker's concurrency limit. -
2 This MR Adds the on_concurrency_limit_deferred and on_concurrency_limit_resumed worker hooks. 1
3 !264410 (merged) Uses the per-user scope for CreatePipelineWorker and MergeRequests::CreatePipelineWorker. 1
4 !267879 (merged) Keeps a pipeline creation request in progress while its job is deferred, using the hooks from step 2. 2

Steps 3 and 4 do not depend on each other and can merge in any order.

How to set up and validate locally

class TestLimitedWorker
  include ApplicationWorker

  concurrency_limit -> { 1 }

  def self.on_concurrency_limit_deferred(args) = Rails.logger.info("deferred #{args}")
  def self.on_concurrency_limit_resumed(args) = Rails.logger.info("resumed #{args}")

  def perform(*) = sleep 30
end
  1. Enable the sidekiq_concurrency_limit_middleware ops flag.
  2. Enqueue TestLimitedWorker 3 times.
  3. Check the log: 2 deferred lines, then a resumed line each time a job finishes.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Furkan Ayhan

Merge request reports

Loading
Loading