MultiStore instances spinning up excessive Redis connections on start-up

To facilitate online Redis migrations, we use MultiStore to wrap over 2 Redis connections and performs dual-writes. However, this means extra connection pools are created with each MultiStore usage.

For instance: SharedState is used twice, once as itself and another time in SidekiqStatus. ClusterSharedState is used 3 times, once as itself and twice more via SharedState.

In the recent migration, we noticed that Redis connections was near saturation due to the extra connections to ServiceRedisClusterSharedState that were spun up on start-up. See production#17134 (comment 1662049021).

Why does this problem only afflict Redis Cluster? In the redis gem, connections to Redis Clusters are not lazy as they will create at least 1 connection on initialization to fetch topology information using CLUSTER NODES and COMMAND. This would create at least 1 Redis connection. As most of our Redis instances are Redis Cluster, excessive use of MultiStore without clean up could lead to such increase in client connections.

Proposals

1. Clean up MultiStore usages

• db load balancing can be cleaned up safely -- gitlab-org/gitlab!137830 (merged)
• sidekiq status can be removed and set to SharedState since the migration terminated a long while back.

2. Borrow the connection from the original instance's connection pool

This would mean instead of creating 3 connection pools with their own Redis connection to ClusterSharedState, the 2 MultiStore will use ClusterSharedState's connection pool using the .with method.\

I have drafted an MR gitlab-org/gitlab!137764 (merged) but in light of the timeline for &1055. Perhaps option 1 would suffice in keeping the connection count low as the connection would now double instead of triple during each deploy.