Skip to content

Allow to interrupt running sidekiq jobs

Kamil Trzciński requested to merge sidekiq-interrupt-running-jobs into master

What does this MR do?

This is Proof-of-Concept. It is concurrent-safe. It is cheap.

This implements the proposal from: https://gitlab.com/gitlab-org/gitlab-ce/issues/51096#note_203681105:

  • This adds a middleware to track all threads for running jobs.
  • This makes sidekiq to watch for redis-delivered notifications.
  • This makes be able to send notification to interrupt running sidekiq jobs.
  • This does not take into account any native code, as Thread.raise generates exception once the control gets back to Ruby.
  • The separate measure should be taken to interrupt gRPC, shellouts, or anything else that escapes Ruby.

How to use it?

jid = Chaos::SleepWorker.perform_async(1000);
sleep 2;
Gitlab::SidekiqStatus::CancelJobs.cancel_job(jid);

Remarks

We don't need any new code to remove jobs from Sidekiq, as this is already provided by Sidekiq::API:

queue = Sidekiq::Queue.new("mailer")
queue.each do |job|
  job.klass # => 'MyWorker'
  job.args # => [1, 2, 3]
  job.delete if job.jid == 'abcdef1234567890'
end

Does this MR meet the acceptance criteria?

Conformity

Performance and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by 🤖 GitLab Bot 🤖

Merge request reports