Draft: Fix code reloading for Sidekiq in development
What does this MR do and why?
We were seeing deadlocks in development because code reloading is not
thread-safe, and we had multiple Sidekiq threads attempting to reload
ApplicationSetting
.
Sidekiq launches heartbeat, scheduler, and job processor threads that
talk to Redis. We have a custom Redis instrumentation middleware that
logs Prometheus metrics. During a code reload, each of those threads
attempt to load ApplicationSetting
to determine whether Prometheus
metrics are enabled.
As described in
https://guides.rubyonrails.org/threading_and_code_execution.html,
Rails automatically adds Rack middlewares in development to ensure
code reloading is thread-safe. However, the Sidekiq threads don't use
these Rack middlewares, so we need to call
Rails.application.executor.wrap
manually.
Relates to:
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
How to set up and validate locally
- Check out this branch.
- Run
gdk restart rails
. - Run
gdk tail rails-background-jobs
in another window. - Make some code changes to GitLab.
- Create a merge request or perform other actions to cause background jobs to run.
- From
bin/rails console
, you could also kick off a job manually:AuthorizedProjectsWorker.perform_async(1)
. - Watch that the log messages in step 3 continue to flow, and ensure merge requests still get refreshed.