redis-cache: send read traffic to replicas
TLDR: We are seeing frequent CPU saturation on redis-cache. This load is primarily from reads. Currently all read traffic is being sent to the master. I propose we use the replicas for read scaling.
Background
The redis-cache
master is hitting CPU saturation. This has been an ongoing problem for a while now.
Sample incidents: production#928 (closed), https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7157, https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/9312, https://gitlab.com/gitlab-com/gl-infra/production/issues/1722, production#1726 (closed), a bunch more.
There's been some analysis and high-level ideas on how to address this:
Prior art
The main proposals so far have been:
- Workload analysis: profiling redis-server and tcpdump traffic capture to reduce load.
- Sharding: separate redises, sharded functionally (gitlab-org/gitlab#29887 (closed)) or via consistent hashing (scalability#49 (closed)).
- Replacing redis: redis cluster being one of the options (&80).
I believe we will want some form of sharding eventually. Especially since it allows us to scale writes and memory.
However, most of those solutions require some investment. Especially if we want to keep our redises HA. Redis cluster seems like a strong candidate, but requires work to get into prod.
Analysis
If we look at the workload, most of the load appears to be coming from reads. Perf analysis also showed a lot of time spent in read and write syscalls (&80).
Here is a sum of time grouped by command, on the master:
That top line is get
commands.
Here is that same query on one of the replicas:
There are no get
s being issued against it at all.
Now, let's look at CPU (red is the master, teal and green are replicas):
The replicas have lots of idle CPU.
Proposal
Since we're not constrained by memory or write load, we don't actually need to shard.
We can send read traffic to the replicas instead. This takes load off the master, and even allows us to scale reads horizontally by adding more nodes.
This also does not require any infra-level changes. We already have a redis sentinel based setup with 2 replicas. So far this has only been used for HA, but we can use it for read scaling too.
A change would need to be made to how we use the redis client (redis-rb in gitlab-org/gitlab). When constructing the sentinel-based client, a role
can be supplied:
redis = Redis.new(url: "redis://mymaster", sentinels: SENTINELS, role: :master)
This can either be :master
or :slave
[sic]. When set to slave, it will connect to a random read-only replica.
We should be able to wrap the redis client passed to ActiveSupport::Cache. This wrapper can then send read commands (get
, smembers
, sismember
, exists
, mget
, hmget
) to a read replica, and everything else to the master.
I believe this could address the CPU saturation issues we've been seeing on redis-cache.
Though there may of course also be issues with this proposal that I didn't anticipate, so I'd love to get some feedback on it. :)