Add custom router for sidekiq sharding

What does this MR do and why?

This MR introduces a custom router for Sidekiq sharding in GitLab rails.

See issue: gitlab-com/gl-infra/scalability#2817

This MR primarily handles the bulk of sidekiq operations in gitlab-rails

• perform_ variants using the Sidekiq::Client.via in a minimally invasive patch
• scheduled jobs using a custom enq
• ability to define external Redis for sidekiq sharding using queues_shard_ prefix in redis.yml and accessible using Gitlab::Redis::Queues.instances
• update metrics and logs to include sidekiq destination shard's redis details

Sidekiq API usages listed in gitlab-com/gl-infra/scalability#2817 (comment 1763832817) will be addressed in a separate MR.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before	After

How to set up and validate locally

Create feature flag definitions since Feature would raise exceptions

➜  gitlab git:(sc1-sidekiq-shard-routing) ✗ cat config/feature_flags/ops/route_to_queues_shard_01.yml
---
name: route_to_queues_shard_01
feature_issue_url:
introduced_by_url:
rollout_issue_url:
milestone: '16.9'
group: group::scalability
type: ops
default_enabled: false

1. There are 2 files to update

config/gitlab.yml

## Sidekiq
  sidekiq:
    log_format: json # (default is also supported)
    routing_rules:
      - ["tags=needs_own_queue", null]
      - ["worker_name=Chaos::SleepWorker", "chaos", "queues_shard_01"]
      - ["worker_name=Chaos::CpuSpinWorker", "chaos", "queues_shard_02"]
      - ["*", "default"]

config/redis.yml

➜  gitlab git:(sc1-sidekiq-sharding-router) ✗ cat config/redis.yml
---
development:
  queues_shard_01:
    url: unix:/Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket?db=3
  queues_shard_02:
    url: unix:/Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket?db=4

2. Open up a rails console gdk rails c

Loading development environment (Rails 7.0.8)
[1] pry(main)> Feature.enable(:enable_sidekiq_shard_router)
=> true
[2] pry(main)> Feature.enable(:sidekiq_route_to_queues_shard_01)
=> true
[3] pry(main)> Feature.enable(:sidekiq_route_to_queues_shard_03)
=> true
[4] pry(main)> Chaos::SleepWorker.perform_async(13)                                                        
=> "9850bb3c74426edbc6986494"
[5] pry(main)> Chaos::CpuSpinWorker.perform_async(13)                                                      
=> "a081124ce04955011893635a"

3. Verify on Redis

➜  gitlab git:(sc1-sidekiq-sharding-router) ✗ gdk redis-cli -n 3 lrange queue:chaos 0 -1
1) "{\"retry\":3,\"queue\":\"chaos\",\"backtrace\":true,\"version\":0,\"store\":\"queues_shard_01\",\"queue_namespace\":\"chaos\",\"args\":[13],\"class\":\"Chaos::SleepWorker\",\"jid\":\"9850bb3c74426edbc6986494\",\"created_at\":1696495293.8569322,\"correlation_id\":\"36c06515ea92e950ce8d5693a919674e\",\"worker_data_consistency\":\"always\",\"idempotency_key\":\"resque:gitlab:duplicate:chaos:c1460b1b4faf1e1c8ff822685001efd84fe12d6dc203ccb3fd79dcf26956b926\",\"duplicate-of\":\"0b2314874ab80a5b8112cd4a\",\"size_limiter\":\"validated\",\"enqueued_at\":1696495293.868097}"
➜  gitlab git:(sc1-sidekiq-sharding-router) ✗ gdk redis-cli -n 4 lrange queue:chaos 0 -1
1) "{\"retry\":3,\"queue\":\"chaos\",\"backtrace\":true,\"version\":0,\"store\":\"queues_shard_02\",\"queue_namespace\":\"chaos\",\"args\":[13],\"class\":\"Chaos::CpuSpinWorker\",\"jid\":\"a081124ce04955011893635a\",\"created_at\":1696495302.608432,\"correlation_id\":\"36c06515ea92e950ce8d5693a919674e\",\"worker_data_consistency\":\"always\",\"idempotency_key\":\"resque:gitlab:duplicate:chaos:12de73968ed841908a3da05c556ade12b35091198373cdd7fa4a73a0409d17c9\",\"size_limiter\":\"validated\",\"enqueued_at\":1696495302.614424}"

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.