Migrate Redis::SharedState to Redis::ClusterSharedState

What does this MR do and why?

This MR adds the MultiStore helper to migrate workloads from Gitlab::Redis::SharedState to Gitlab::Redis::ClusterSharedState.

MultiStore helps us to perform double writes from the old store (secondary) to the new store (primary) and cutover reads to the primary store. See https://docs.gitlab.com/ee/development/redis/new_redis_instance.html#proposed-solution-migrate-data-by-using-multistore-with-the-fallback-strategy for more information.

Issue link: gitlab-com/gl-infra/scalability#2588 (closed)

How to set up and validate locally

1. Setup gdk with redis cluster https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/redis_cluster.md
2. Ensure ClusterSharedState uses Redis Cluster

❯ cat config/redis.yml
development:
  cluster_shared_state:
    cluster:
      - redis://localhost:6000

test:
  cluster_shared_state:
    cluster:
      - redis://localhost:6000

3. Check in console:

[9] pry(main)> Gitlab::Redis::ClusterSharedState.store
=> #<Redis client v4.8.0 for redis://127.0.0.1:6000/0 redis://127.0.0.1:6001/0 redis://127.0.0.1:6002/0>
[10] pry(main)> Gitlab::Redis::SharedState.store
=> #<Redis client v4.8.0 for unix:///Users/gregoriusmarco/Documents/workspace/gdk-10-22/redis/redis.socket/0>

4. Without any of the FF enabled:

[12] pry(main)> Gitlab::Redis::SharedState.with { |r| r.flushdb }
=> "OK"
[14] pry(main)> Gitlab::Redis::SharedState.with { |r| r.set("foo", "bar") }
=> "OK"
[16] pry(main)> Gitlab::Redis::SharedState.with { |r| r.get("foo") }
=> "bar"

5. At this point, check that foo only exists in SharedState:

❯ gdk redis-cli -n 0 get foo
"bar"
❯ redis-cli -p 6000 get foo
(error) MOVED 12182 127.0.0.1:6002
❯ redis-cli -p 6002 get foo
(nil)

6. Enable double write FF:

[17] pry(main)> Feature.enable(:use_primary_and_secondary_stores_for_shared_state)
=> true
[18] pry(main)> Gitlab::Redis::SharedState.with { |r| r.get("foo") }
=> "bar"
[19] pry(main)> Gitlab::Redis::SharedState.with { |r| r.set("hello", "world") }
=> "OK"
[21] pry(main)> Gitlab::Redis::SharedState.with { |r| r.get("hello") }
=> "world"

7. Check that hello also exists in both SharedState and ClusterSharedState:

❯ gdk redis-cli -n 0 get hello
"world"
❯ redis-cli -p 6000 get hello
"world"

8. Enable FF to set ClusterSharedState as default_store. This means we are only reading from ClusterSharedState, still double-writing to ClusterSharedState and SharedState.

[22] pry(main)> Feature.enable(:use_primary_store_as_default_for_shared_state)
=> true
[23] pry(main)> Gitlab::Redis::ClusterSharedState.with{ |r| r.set("bababa", "nana") } # only set the key in ClusterSharedState
=> "OK"
[24] pry(main)> Gitlab::Redis::SharedState.with{ |r| r.get("bababa") } # we are stil able to get "bababa"
=> "nana"

9. Check in redis:

❯ gdk redis-cli -n 0 get bababa
(nil)
❯ redis-cli -p 6002 get bababa
"nana"

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.