Draft: Transit to non-namespaced keys in Sidekiq

What does this MR do and why?

Refer to gitlab-com/gl-infra/scalability#1982 (closed) for motivations and discussions

This MR adds several patches which feature-flag-gated logic to switch from resque:gitlab namespace to non-namespaced keys in sidekiq.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

1. Create dummy worker In a file named app/workers/fake_worker.rb, create.

# frozen_string_literal: true

class FakeWorker
  include ApplicationWorker

  data_consistency :always
  idempotent!

  def perform(duration, delay = nil)
    if delay.nil?
      FakeWorker.perform_async(duration)
    else
      FakeWorker.perform_at(delay, duration, delay)
    end

    sleep(duration)
  end
end

2. Restart gdk

gdk restart

3. Observe states pre-feature-flag toggle by starting up a redis-cli session

gdk redis-cli -n 1
> keys * // should see all keys in resque:gitlab namespace

4. Trigger dummy jobs

gdk rails c
[1] pry(main)> (1..20).each {|n| FakeWorker.perform_async(n)}
[2] pry(main)> (1..20).each {|n| FakeWorker.perform_async(n)} // create duplicates
[3] pry(main)> (1..20).each {|n| FakeWorker.perform_at(n, n, n)}

Check llen resque:gitlab:queue:default and llen <working queue>. The name of the working queue can be retrieved via keys *working*.

5. Toggle feature flag

# within gdk rails c
Feature.enable(:disable_sidekiq_namespace, true)

6. Observe migration of jobs

• Queues: Compare llen resque:gitlab:queue:default vs llen queue:default, the former should reduce to 0 while the latter start populating.
• ScheduleSet: Compare zcard resque:gitlab:schedule vs zcard schedule, the former should stop changing while the latter is populatled.

Other keys to observe

• processes should contain the new process
• cron_jobs and respective cron_jobs:* should be migrated

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.