Skip to content

Update multistore to read from default

Sylvester Chin requested to merge schin1-multistore-read-default into master

What does this MR do and why?

This MR modifies MultiStore to read from default_store with fallbacks to fallback_store instead of always reading from primary_store and falling back to secondary_store.

In gitlab-com/gl-infra/scalability#1994 (comment 1170461215) and later on in gitlab-com/gl-infra/scalability#2161 (comment 1272903843), the existing read behaviour could lead to stale reads during rollout which is not ideal for performance-sensitive workloads. Ideally we want to continue reading from redis-cache until the migration target is fully warmed up.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

How to set up and validate locally

Using gdk rails c

[2] pry(main)> primary_store = ::Redis.new(Gitlab::Redis::SharedState.params)
=> #<Redis client v4.8.0 for unix:///Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket/0>
[3] pry(main)> secondary_store = ::Redis.new(Gitlab::Redis::Queues.params)
=> #<Redis client v4.8.0 for unix:///Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket/1>
[4] pry(main)> mm = Gitlab::Redis::MultiStore.new(primary_store, secondary_store, 'sidekiq_status')
=> #<Gitlab::Redis::MultiStore:0x0000000157654458
 @instance_name="sidekiq_status",
 @primary_store=#<Redis client v4.8.0 for unix:///Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket/0>,
 @same_redis_store=false,
 @secondary_store=#<Redis client v4.8.0 for unix:///Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket/1>>
[5] pry(main)> mm.default_store
=> #<Redis client v4.8.0 for unix:///Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket/0>
[6] pry(main)> mm.fallback_store
=> #<Redis client v4.8.0 for unix:///Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket/1>
[7] pry(main)> Feature.disable(:use_primary_store_as_default_for_sidekiq_status)
=> true
[8] pry(main)> mm.default_store
=> #<Redis client v4.8.0 for unix:///Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket/1>
[9] pry(main)> mm.fallback_store
=> #<Redis client v4.8.0 for unix:///Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket/0>

To verify read behaviour,

[6] pry(main)> Feature.disable(:use_primary_store_as_default_for_sidekiq_status)
[9] pry(main)> primary_store.set('x', 1)
=> "OK"
[10] pry(main)> secondary_store.set('x', 2)
=> "OK"
// read from secondary store which is the default
[11] pry(main)> mm.get('x')
=> "2"

// read from primary store which is the default now
[12] pry(main)> Feature.enable(:use_primary_store_as_default_for_sidekiq_status)
=> true
[13] pry(main)> mm.get('x')
=> "1"

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Sylvester Chin

Merge request reports