Fix code reloading for Sidekiq in development
What does this MR do and why?
We have been seeing deadlocks in development because code reloading is
not thread-safe, since we have multiple Sidekiq threads attempting to
reload ApplicationSetting
after it is unloaded.
Sidekiq launches heartbeat, scheduler, and job processor threads that
talk to Redis. We have a custom Redis instrumentation middleware that
logs Prometheus metrics. Each of those those threads attempt to load
ApplicationSetting
to determine whether Prometheus metrics are
enabled, but this behavior is not thread-safe.
As described in
https://guides.rubyonrails.org/threading_and_code_execution.html, Rails
automatically adds Rack middlewares in development to ensure
thread-safety during code reloading. However, the Sidekiq threads don't
use these Rack middlewares. We'd have to call
Rails.application.executor.wrap
manually around every Redis call, but
that introduces too much locking overhead.
Instead, let's just disable the RedisClient
instrumentation when
code reloading is enabled since it doesn't play well with it.
Relates to:
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
How to set up and validate locally
How to set up and validate locally
- Check out this branch.
- Run
gdk restart rails
. - Run
gdk tail rails-background-jobs
in another window. - Make some code changes to GitLab:
git checkout HEAD~12
, for example. - Create a merge request or perform other actions to cause background jobs to run.
- From
bin/rails console
, you could also kick off a job manually:AuthorizedProjectsWorker.perform_async(1)
. - Watch that the log messages in step 3 continue to flow, and ensure merge requests still get refreshed.