MultiStore instances spinning up excessive Redis connections on start-up
To facilitate online Redis migrations, we use MultiStore
to wrap over 2 Redis connections and performs dual-writes. However, this means extra connection pools are created with each MultiStore
usage.
For instance: SharedState
is used twice, once as itself and another time in SidekiqStatus
. ClusterSharedState
is used 3 times, once as itself and twice more via SharedState
.
In the recent migration, we noticed that Redis connections was near saturation due to the extra connections to ServiceRedisClusterSharedState that were spun up on start-up. See production#17134 (comment 1662049021).
Why does this problem only afflict Redis Cluster? In the redis
gem, connections to Redis Clusters are not lazy as they will create at least 1 connection on initialization to fetch topology information using CLUSTER NODES
and COMMAND
. This would create at least 1 Redis connection. As most of our Redis instances are Redis Cluster, excessive use of MultiStore without clean up could lead to such increase in client connections.
Proposals
- Clean up MultiStore usages
- db load balancing can be cleaned up safely -- gitlab-org/gitlab!137830 (merged)
- sidekiq status can be removed and set to
SharedState
since the migration terminated a long while back.
- Borrow the connection from the original instance's connection pool
This would mean instead of creating 3 connection pools with their own Redis connection to ClusterSharedState
, the 2 MultiStore
will use ClusterSharedState
's connection pool using the .with
method.\
I have drafted an MR gitlab-org/gitlab!137764 (merged) but in light of the timeline for &1055 (closed). Perhaps option 1 would suffice in keeping the connection count low as the connection would now double instead of triple during each deploy.