Update multistore to read from default
What does this MR do and why?
This MR modifies MultiStore
to read from default_store
with fallbacks to fallback_store
instead of always reading from primary_store
and falling back to secondary_store
.
In gitlab-com/gl-infra/scalability#1994 (comment 1170461215) and later on in gitlab-com/gl-infra/scalability#2161 (comment 1272903843), the existing read behaviour could lead to stale reads during rollout which is not ideal for performance-sensitive workloads. Ideally we want to continue reading from redis-cache
until the migration target is fully warmed up.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
How to set up and validate locally
Using gdk rails c
[2] pry(main)> primary_store = ::Redis.new(Gitlab::Redis::SharedState.params)
=> #<Redis client v4.8.0 for unix:///Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket/0>
[3] pry(main)> secondary_store = ::Redis.new(Gitlab::Redis::Queues.params)
=> #<Redis client v4.8.0 for unix:///Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket/1>
[4] pry(main)> mm = Gitlab::Redis::MultiStore.new(primary_store, secondary_store, 'sidekiq_status')
=> #<Gitlab::Redis::MultiStore:0x0000000157654458
@instance_name="sidekiq_status",
@primary_store=#<Redis client v4.8.0 for unix:///Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket/0>,
@same_redis_store=false,
@secondary_store=#<Redis client v4.8.0 for unix:///Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket/1>>
[5] pry(main)> mm.default_store
=> #<Redis client v4.8.0 for unix:///Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket/0>
[6] pry(main)> mm.fallback_store
=> #<Redis client v4.8.0 for unix:///Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket/1>
[7] pry(main)> Feature.disable(:use_primary_store_as_default_for_sidekiq_status)
=> true
[8] pry(main)> mm.default_store
=> #<Redis client v4.8.0 for unix:///Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket/1>
[9] pry(main)> mm.fallback_store
=> #<Redis client v4.8.0 for unix:///Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket/0>
To verify read behaviour,
[6] pry(main)> Feature.disable(:use_primary_store_as_default_for_sidekiq_status)
[9] pry(main)> primary_store.set('x', 1)
=> "OK"
[10] pry(main)> secondary_store.set('x', 2)
=> "OK"
// read from secondary store which is the default
[11] pry(main)> mm.get('x')
=> "2"
// read from primary store which is the default now
[12] pry(main)> Feature.enable(:use_primary_store_as_default_for_sidekiq_status)
=> true
[13] pry(main)> mm.get('x')
=> "1"
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.