Fix sidekiq status primary store definition

What does this MR do and why?

This MR fixes sidekiq status primary store definition. As Gitlab::Redis::SharedState is a multistore, the primary store in Gitlab::Redis::SidekiqStatus cannot be defined using SharedState's params.

Contributes to gitlab-com/gl-infra/scalability#2588 (closed)

Context: the bug was discovered during a gstg migration (gitlab-com/gl-infra/production#17006 (comment 1643038057)).

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before	After

How to set up and validate locally

On master branch

[3] pry(main)> Gitlab::Redis::SidekiqStatus.with {|c| c}
=> #<Gitlab::Redis::MultiStore:0x0000000125de7d50
 @instance_name="SidekiqStatus",
 @primary_store=
  #<Redis client v4.8.0 for unix:///Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket/0>,
 @same_redis_store=false,
 @secondary_store=
  #<Redis client v4.8.0 for unix:///Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket/1>>

On this branch

[1] pry(main)> Gitlab::Redis::SidekiqStatus.with {|c| c}
=> #<Gitlab::Redis::MultiStore:0x00000001507e5f08
 @instance_name="SidekiqStatus",
 @primary_store=
  #<Gitlab::Redis::MultiStore:0x0000000150947518
   @instance_name="SharedState",
   @primary_store=
    #<Redis client v4.8.0 for redis://127.0.0.1:6000/0 redis://127.0.0.1:6001/0 redis://127.0.0.1:6002/0>,
   @secondary_store=
    #<Redis client v4.8.0 for unix:///Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket/0>>,
 @secondary_store=
  #<Redis client v4.8.0 for unix:///Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket/1>>

Another way of validation is

1. Set up config/redis.yml and enable redis_cluster in gdk using https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/data/announcements/0004_redis_cluster_support.yml

➜  gitlab git:(master) ✗ cat config/redis.yml

---
development:
  cluster_shared_state:
    cluster:
      - "redis://localhost:6001"
      - "redis://localhost:6002"
      - "redis://localhost:6000"

2. On master branch, in a rails console:

Feature.enable(:use_primary_and_secondary_stores_for_shared_state)
Feature.enable(:use_primary_and_secondary_stores_for_sidekiq_status)

Gitlab::Redis::SidekiqStatus.with {|c| c.set('testkey', 1)}

Check all 3 Redis stores, the Redis cluster would not have testkey written to it.

3. On this branch

Feature.enable(:use_primary_and_secondary_stores_for_shared_state)
Feature.enable(:use_primary_and_secondary_stores_for_sidekiq_status)

Gitlab::Redis::SidekiqStatus.with {|c| c.set('testkey2', 1)}

Check all 3 stores to verify that the key is written to all 3.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.