Enable sidekiq to fetch from namespaced and non-namespaced queues

What does this MR do and why?

This MR adds the ability to transition away from Redis::Namespace usage within Sidekiq since namespace support is dropped in Sidekiq 7.0.

It does so via 2 changes

1. updates the sidekiq reliable fetcher with 2 option

• namespace: to poll from both namespaced and non-namespaced queues
• alternative_store: an alternative Redis connection pool instead of Sidekiq.redis

2. uses 2 environment variables to toggle:

• SIDEKIQ_POLL_NON_NAMESPACED adds Gitlab::Redis::Queues as an alternative store and defines a namespace. This lets the poller poll from both namespaced and non-namespaced keys
• SIDEKIQ_ENQUEUE_NON_NAMESPACED configures Sidekiq.redis to not use a namespace

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

How to set up and validate locally

1. Run gdk restart rails

Check that keys are namespaced with resque:gitlab: using keys * in redis-cli

2. Enqueue a worker in non-namespaced queue using a rails console that has SIDEKIQ_ENQUEUE_NON_NAMESPACED envvar set to true

➜  gitlab git:(sc1-skfetcher-dual-fetch) ✗ SIDEKIQ_ENQUEUE_NON_NAMESPACED=true gdk rails c
--------------------------------------------------------------------------------
 Ruby:         ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [arm64-darwin22]
 GitLab:       16.3.0-pre (f9ac1eda7c9) EE
 GitLab Shell: 14.23.0
 PostgreSQL:   13.9
--------------------------------------------------------------------------------
Loading development environment (Rails 7.0.6)
[1] pry(main)> Chaos::SleepWorker.perform_async(100)
=> "617470d9c0ca2d41584dea01"

3. Confirm that the job is not picked up using redis-cli

➜  gitlab git:(sc1-skfetcher-dual-fetch) ✗ gdk redis-cli -n 1
redis /Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket[1]> lrange "queue:default" 0 -1
1) "{\"retry\":3,\"queue\":\"default\",\"backtrace\":true,\"version\":0,\"queue_namespace\":\"chaos\",\"args\":[100],\"class\":\"Chaos::SleepWorker\",\"jid\":\"617470d9c0ca2d41584dea01\",\"created_at\":1691397251.340776,\"correlation_id\":\"e6c292659ebe491c4bd88e4ceaf6c328\",\"worker_data_consistency\":\"always\",\"idempotency_key\":\"resque:gitlab:duplicate:default:b636855eb030c46b2e40c1c07d1955780f5af4b288f6dc4c0451b6a609dcb24d\",\"size_limiter\":\"validated\",\"enqueued_at\":1691397251.351478}"

4. Toggle SIDEKIQ_POLL_NON_NAMESPACED feature-flag in GDK's env.runit file

export SIDEKIQ_POLL_NON_NAMESPACED=true

5. Restart gdk using gdk restart rails

6. Check that job is consumed after a while:

redis /Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket[1]> lrange "queue:default" 0 -1
(empty array)

Check the logs using:

cat log/sidekiq.log | rg Chaos
cat log/sidekiq.log | rg Chaos | rg 9a2315d9b0bd3f21da79c03 # filter using the jid for finer grain search

There should be 2 log line (1 for when the job is picked up and another when the jobs is completed) with ~100s apart.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.