Skip to content

Draft: Fix code reloading for Sidekiq in development

Stan Hu requested to merge sh-fix-code-reload-redis-middleware into master

What does this MR do and why?

We were seeing deadlocks in development because code reloading is not thread-safe, and we had multiple Sidekiq threads attempting to reload ApplicationSetting.

Sidekiq launches heartbeat, scheduler, and job processor threads that talk to Redis. We have a custom Redis instrumentation middleware that logs Prometheus metrics. During a code reload, each of those threads attempt to load ApplicationSetting to determine whether Prometheus metrics are enabled.

As described in https://guides.rubyonrails.org/threading_and_code_execution.html, Rails automatically adds Rack middlewares in development to ensure code reloading is thread-safe. However, the Sidekiq threads don't use these Rack middlewares, so we need to call Rails.application.executor.wrap manually.

Relates to:

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

  1. Check out this branch.
  2. Run gdk restart rails.
  3. Run gdk tail rails-background-jobs in another window.
  4. Make some code changes to GitLab.
  5. Create a merge request or perform other actions to cause background jobs to run.
  6. From bin/rails console, you could also kick off a job manually: AuthorizedProjectsWorker.perform_async(1).
  7. Watch that the log messages in step 3 continue to flow, and ensure merge requests still get refreshed.
Edited by Stan Hu

Merge request reports