Redis Cluster performance measurements

Following on from #1265 (closed) and as part of our general Redis Strategy (&557), we will create a test harness in which we can test Redis, Redis-Cluster, and any other candidates (KeyDB) against:

1. Current production level workloads, with realistic key names, data sizes, and access patterns derived from real production traffic (but anonymized/faked data)
2. Scaled-up production level workloads that assumes equal growth across all existing traffic

This will inform us:

1. How much scalability we currently have with standalone Redis, from which we could probably estimate some calendar time-frames based on current growth,
2. Whether Redis Cluster is a reasonable option once we approach those critical thresholds (i.e. key hashing distributes well, with no odd hotspots working against us, or other gotchas),
3. How much scalability Redis Cluster will give us (does it match the theory of N-times scaling from N primary members)
4. If any other candidates are even worth considering (expectation: probably not, but this should make that decision clear based on evidence).

Limitations

It is acknowledged that the predictive power of this is limited to increases on existing traffic, and new functionality being added to the application or access patterns changing will reduce that power. However it should result in tooling/documented procedures that can be re-run trivially in future as necessary.

Scope

We currently have 5 active Redis instances, being persistent, cache, sidekiq, rate-limiting, and tracechunks, with a 6th (sessions) on the way. Sidekiq is known to be incompatible with Redis Cluster in some fundamental ways and is therefore out-of-scope. Tracechunks is not CPU constrained in any way (peaks <= 5% CPU saturation) and is unlikely to be so before other limits are hit (network traffic), and is therefore likewise also out of scope. Therefore this issue will cover persistent, cache, and rate-limiting, and will be complete before the sessions work is far enough progressed to be able to extract the required information. We'll try to give consideration to that (sessions) during analysis, but otherwise we'll have to extrapolate a little.

Steps

For each instance:

• Collect production peak traffic (tcpdump, probably) and extract the operations, keynames, data sizes, and access patterns, particularly write/read/update/delete sequences to try and match correctness rather than just individual operations.
• Write a tool that can take a high-level data description of the traffic patterns obtained in the first step and generate traffic against a Redis instance at current production-level volumes. Verify that it results in similar CPU saturation to production, and adjust as necessary until it comes close. +/- 5% is probably sufficient.
• Run this tool at higher volumes to determine how much more our current Redis design can take before it melts
• Run this tool at higher volumes against the other target configurations (below) to determine their breaking limits.

Target configurations

1. As current, a standalone Redis primary with 2 replicas
2. KeyDB - a standalone primary with 2 replicas
3. Base Redis Cluster: 3 members
4. Larger Redis Clusters: 5 members, 10 members. That's sufficient variants unless the data shows any significant non-linearity that needs more inspection.

For all Redis Cluster tests, there will be 2 replicas per primary; in practice I expect we will be wanting to have the same zone resilience as we have now (be able to lose an entire zone and still have a primary + replica), so we need to ensure that we're measuring any impact from that replication traffic.

Effort estimate

In the order of 5-7 working days.

Key Results

Persistent Redis (SharedState)

• #1346 (comment 721668514) - on current load patterns, we have perhaps 10% growth headroom before life gets interesting, and less than 25% before disaster.
• #1346 (comment 721672696) - partitioning Session handling to its own Redis (here) could drop CPU saturation from 90% to 70%, and would give us more like 35% headroom.
• #1346 (comment 721679765) - partitioning session handling and CI pubsub (key workhorse:notifications) would give 100% headroom (2x scaling
• #1346 (comment 722995035)
  • A 3x Redis-Cluster, with session and CI pubsub partitioned would give 5x scaling with some moderate remaining headroom (6x is probably a push)
  • CI pubsub uses something like 20% absolute CPU on a single Redis node even if clustered (and with no additional workhorse listeners; it gets worse as we add more which will happen with time), because it passes through a single key which hashes to a single node. While not technically incompatible with Cluster, this means it is a significant constraint, leaving spare CPU on the table on the other nodes that are not handling that traffic. Therefore partitioning it to its own Redis would be the sensible mid-term choice if we wanted to move to Cluster. We could defer it from the short term in terms of practical CPU usage given current traffic. Or: it's a matter of cost in the short-to-medium term and we can probably do the math on spend for more nodes vs spend on engineer time to partition, at the appropriate time.
• #1346 (comment 723011504) - As we expand the Cluster size, the capacity for traffic grows basically linearly with number of nodes, which is in line with the documentation. There was some non-linearity in moving to cluster (some basic overhead), but that is rapidly eclipsed by the node scaling as we add nodes.

Cache

#1346 (comment 724728525) - the basic load generator functions, but will need tuning before actual reliable results can be extracted.

Ratelimiting

#1346 (comment 724725842) - the basic load generator creates production level loads under known configuration. It has not been tested to destruction on non-Cluster, nor against Redis Cluster at all, but gives every indication (as expected) of basically linear scaling of traffic to CPU. This will need to be picked up again in future to complete.