Skip to content

Re-use Redis instance's connection pool in MultiStore

Sylvester Chin requested to merge sc1-multistore-conn-pool-sharing into master

What does this MR do and why?

This MR reworks MultiStore to use the connection pools in the Redis instances instead of creating new connections.

Issue: gitlab-com/gl-infra/scalability#2646 (closed)

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

  1. Set up your config/redis.yml
➜  gitlab git:(sc1-multistore-conn-pool-sharing) ✗ cat config/redis.yml
---
development:
  cluster_shared_state:
    cluster:
      - "redis://localhost:6001"
      - "redis://localhost:6002"
      - "redis://localhost:6000"
  1. Run the following snippet in a Rails console:
Feature.enable(:use_primary_and_secondary_stores_for_shared_state)
Feature.enable(:use_primary_and_secondary_stores_for_sidekiq_status)

Gitlab::Redis::SharedState.with {|c| c.pfadd 'abc', 1 }
Gitlab::Redis::ClusterSharedState.with {|c| c.pfadd 'abc', 1 }
  1. Run the following on your terminal. You should only see 1 connection as all 2 Redis wrapper classes will get the connection to the Redis Cluster using ClusterSharedState's connection pool.
redis-cli -p 6001 client list | rg pfadd
  1. Repeat step 2 on master branch, you should see 2 connections as each of the 2 Redis wrapper classes spin their own Redis connections

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Sylvester Chin

Merge request reports