Reduce Redis sharding toil
In this meta issue I want to think out loud about some ideas to reduce the toil needed to add Redis shards (like in &857 (closed)). I got these ideas from watching this conference talk and talking to @andrewn and @smcgivern.
The main goal (or dream) is to get to a point where we don't need a cross-functional team (SRE + Backend Engineer) anymore to do a partitioning exercise, but it could be done by an SRE-only team instead.
- Redis shards should be defined as key prefixes
- The Redis client in gitlab-org/gitlab should route requests to the right Redis based on the keys
- Redis shards should be defined in a config file, not in application code
- #2064 Using MultiStore should be a config setting: it is used only during the migration
-
#2063 (closed) There should be just 1 config file
redis.yml
for all Redis shards, instead of one file per shard (redis.cache.yml
,redis.queues.yml
, etc.) - GDK, Omnibus and Helm should not know which shards exist so that new shards don't require GDK and Omnibus and Helm changes
For example, in the config/redis.yml
config file we would see something like this:
db_load_balancing: # shard name
production: # Rails env
prefix: database-load-balancing/ # keys that make up the shard
parent: shared_state # keys used to be on shared_state
url: redis://mymaster # connection details of the shard
mirroring: parent_active # use Gitlab::Redis::MultiStore to mirror data
This causes a new Ruby class Gitlab::Redis::DbLoadBalancing
to be
created when the application boots. The application would still be
using Gitlab::Redis::SharedState
but calls involving keys starting
with database-load-balancing/
would automagically go to
Gitlab::Redis::DbLoadBalancing
.
Some of these things are more difficult than others. I think that (5) and (6) are low risk and immediately valuable in the context of &857 (closed).