Skip to content

Reduce Redis sharding toil

In this meta issue I want to think out loud about some ideas to reduce the toil needed to add Redis shards (like in &857). I got these ideas from watching this conference talk and talking to @andrewn and @smcgivern.

The main goal (or dream) is to get to a point where we don't need a cross-functional team (SRE + Backend Engineer) anymore to do a partitioning exercise, but it could be done by an SRE-only team instead.

  1. Redis shards should be defined as key prefixes
  2. The Redis client in gitlab-org/gitlab should route requests to the right Redis based on the keys
  3. Redis shards should be defined in a config file, not in application code
  4. #2064 (closed) Using MultiStore should be a config setting: it is used only during the migration
  5. #2063 (closed) There should be just 1 config file redis.yml for all Redis shards, instead of one file per shard (redis.cache.yml, redis.queues.yml, etc.)
  6. GDK, Omnibus and Helm should not know which shards exist so that new shards don't require GDK and Omnibus and Helm changes

For example, in the config/redis.yml config file we would see something like this:

db_load_balancing:                    # shard name
  production:                         # Rails env
    prefix: database-load-balancing/  # keys that make up the shard
    parent: shared_state              # keys used to be on shared_state
    url: redis://mymaster             # connection details of the shard
    mirroring: parent_active          # use Gitlab::Redis::MultiStore to mirror data

This causes a new Ruby class Gitlab::Redis::DbLoadBalancing to be created when the application boots. The application would still be using Gitlab::Redis::SharedState but calls involving keys starting with database-load-balancing/ would automagically go to Gitlab::Redis::DbLoadBalancing.

Some of these things are more difficult than others. I think that (5) and (6) are low risk and immediately valuable in the context of &857.

Edited by Jacob Vosmaer