Skip to content

Draft: Patch mget in Redis::Cluster for cross-slot safety and efficiency

Sylvester Chin requested to merge schin1-override-mget-pipeline into master

What does this MR do and why?

This MR patches the mget command to send pipeline gets to the right node and reconstructs the responses to match the input keys ordering.

As discussed in !104030 (closed), the alternative to breaking up a single mget command into multiple get is to optimise it by pipelining the get to each node. This means our multiplicative increase is bounded by the number of redis shards/master-nodes rather than the command's argument length.

This MR includes a CacheCluster which reads from a Redis Cluster.

See gitlab-com/gl-infra/scalability#2004 (closed)

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

Example below:

  1. Set up a local redis cluster

Use config config/redis.cache_cluster.yml

development:
  url: "redis://127.0.0.1:7001"
  cluster:
    - "redis://127.0.0.1:7001"
    - "redis://127.0.0.1:7101"
    - "redis://127.0.0.1:7201"
Click to expand 
#!/bin/bash

set -efo pipefail

COLOR='\033[1;36m'
NC='\033[0m' # No Color
function info {
  echo -e "${COLOR}$@${NC}"
}

killall redis-server || true
find . -type d -name '7*' | parallel rm -r

(parallel -j0 --lb --tag 'mkdir -p {} && cp redis.conf {} && sd 7000 {} {}/redis.conf && cd {} && redis-server redis.conf' ::: 700{1,2,3} 710{1,2,3} 720{1,2,3}) &
sleep 1


redis-cli -p 7001 CLUSTER MEET 127.0.0.1 7002
redis-cli -p 7001 CLUSTER MEET 127.0.0.1 7003

sleep 5

redis-cli -p 7002 CLUSTER REPLICATE "$(redis-cli -p 7001 --raw CLUSTER MYID)"
redis-cli -p 7003 CLUSTER REPLICATE "$(redis-cli -p 7001 --raw CLUSTER MYID)"


redis-cli -p 7001 CLUSTER MEET 127.0.0.1 7101
redis-cli -p 7001 CLUSTER MEET 127.0.0.1 7102
redis-cli -p 7001 CLUSTER MEET 127.0.0.1 7103

sleep 5

redis-cli -p 7102 CLUSTER REPLICATE "$(redis-cli -p 7101 --raw CLUSTER MYID)"
redis-cli -p 7103 CLUSTER REPLICATE "$(redis-cli -p 7101 --raw CLUSTER MYID)"
redis-cli -p 7001 CLUSTER MEET 127.0.0.1 7201
redis-cli -p 7001 CLUSTER MEET 127.0.0.1 7202
redis-cli -p 7001 CLUSTER MEET 127.0.0.1 7203

sleep 5

redis-cli -p 7202 CLUSTER REPLICATE "$(redis-cli -p 7201 --raw CLUSTER MYID)"
redis-cli -p 7203 CLUSTER REPLICATE "$(redis-cli -p 7201 --raw CLUSTER MYID)"

sleep 5

redis-cli -p 7001 CLUSTER ADDSLOTS $(seq 0 16383)

sleep 5

redis-cli --cluster rebalance 127.0.0.1:7001 --cluster-use-empty-masters
sleep 5
redis-cli --cluster info 127.0.0.1:7001
  1. Seed data into redis cluster
[1] pry(main)> Gitlab::Redis::CacheCluster.with {|r| r.set("a", 'a')}
=> "OK"
[2] pry(main)> Gitlab::Redis::CacheCluster.with {|r| r.set("b", 'b')}
=> "OK"
[3] pry(main)> Gitlab::Redis::CacheCluster.with {|r| r.set("c", 'c')}
=> "OK"
[4] pry(main)> Gitlab::Redis::CacheCluster.with {|r| r.set("d", 'd')}
=> "OK"
  1. Read data using mget 
[5] pry(main)> Gitlab::Instrumentation::RedisClusterValidator.allow_cross_slot_commands do
[5] pry(main)*   Gitlab::Redis::CacheCluster.with {|r| r.mget("a", "b", "c", "d")}
[5] pry(main)* end
=> ["a", "b", "c", "d"]
  1. Same setup works with Rails.cache 
[1] pry(main)> Rails.cache.write("a", "a")
=> "OK"
[2] pry(main)> Rails.cache.write("b", "b")
=> "OK"
[3] pry(main)> Rails.cache.write("c", "c")
=> "OK"
[4] pry(main)> Rails.cache.read_multi("a", "b", "c")
=> {"a"=>"a", "b"=>"b", "c"=>"c"}

Note that a, b, c all in different keyslots and nodes.

127.0.0.1:7001> cluster keyslot a
(integer) 15495
127.0.0.1:7001> cluster keyslot b
(integer) 3300
127.0.0.1:7001> cluster keyslot c
(integer) 7365

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Sylvester Chin

Merge request reports