Skip to content

Functional Partitioning for Redis persistent

Proposal

This proposal is based on the Redis Scalability Strategy and presents one of the options for how to scale our redis-persistent set.

Persistent Proposal 1: Application-level partitioning with more Redis-Persistent instances

graph TD
    Rails --> RedisPersist001[Redis Persist 01 'Default']
    Rails --> RedisPersist002[Redis Persist 02 'Pipelines']
    Rails --> RedisPersist003[Redis Persist 03 'Etag Caching, Long Polling']

With this approach, we just continue to carve off vertical slices from the Redis persist instance, moving keyspaces over to new replica sets. For example, we could declare that all Rails Session State will move to a new cluster, then all exclusive locking, etc.

The problem with this approach is that it feels that we have completed the natural division of the clusters, having divided them according to workload, rather than arbitrary divisions. Thus Redis is divided up by cache data, queuing, and persistent storage. The strategy of division by workload is now exhausted and further division would need to be along more arbitrary lines, such as feature category.

Additionally, communicating these divisions with application developers would be overhead, and it would be difficult to ensure that the appropriate servers were used. Keeping consistency would be difficult as the number of connections grows. Hotspot areas would need to be further divided, adding more complexity.

Pros

  • Continuation of the existing scaling strategy - easy to understand

Cons

  • Extra cognitive overhead for developers. “Which connection must I use?”
  • Difficult to verify that the correct Redis is being used for a particular keyset.
  • Hotspots in certain keyspaces are difficult to resolve.

Analysis

The analysis on our near-miss redis saturation event shows heavy key patterns by command count.

$ cat trace.txt | awk '{ print $3, $5 }' | sort | uniq -c | sort -nr | head -n 20

101858 get "database-load-balancing/write-location/user/$NUMBER"
100481 setex "session:gitlab:$NUMBER::$LONGHASH"
98819 get "session:gitlab:$NUMBER::$LONGHASH"
94397 setex "session:user:gitlab:$NUMBER:$NUMBER::$LONGHASH"
94392 sadd "session:lookup:user:gitlab:$NUMBER"
59688 get "database-load-balancing/write-location/build/$NUMBER"
53219 get "runner:build_queue:$PATTERN"
50058 get "etag:$PATTERN"
37630 exists "sidekiq:cancel:$PATTERN"
34510 set "gitlab:exclusive_lease:$PATTERN"
32973 set "etag:$PATTERN"
29712 get "database-load-balancing/write-location/runner/$PATTERN"
22066 strlen "gitlab:ci:trace:$NUMBER:chunks:$NUMBER"
20613 get "database-load-balancing/write-location/ci/build/trace/$NUMBER"
10176 get "projects/$NUMBER/pushes_since_gc"
9976 exists "gitlab:ci:trace:$NUMBER:watched"
9642 set "runner:build_queue:$PATTERN"
8758 pfadd "i_{code_review}_user_vs_code_api_request-$NUMBER-$NUMBER"
8758 expire "i_{code_review}_user_vs_code_api_request-$NUMBER-$NUMBER"
7594 mget "session:gitlab:$NUMBER::$LONGHASH"

Note that this is only based on a 30 second sample, so we may want to do some analysis on longer samples (especially given seasonality on the minute, hour, day, week).

It also is quite difficult to correlate specific patterns with CPU utilization. So this will only be an approximation.

But it provides a good first data point on which key patterns are producing load. This can inform keys that we may want to move out into a separate set of redis nodes.

Some initial ideas (up for debate especially with more data):

  • database-load-balancing/write-location
  • session:gitlab:$NUMBER::$LONGHASH
  • etag:$PATTERN
  • sidekiq:cancel:$PATTERN
  • workhorse:notifications

Migration

If we do proceed this way, we will need to define a migration path.

One way of doing this migration would be by introducing two feature flags. One that enables double-write (write to persistent and the new cluster), and a second one that shifts reads to the new cluster.

Depending on the keys we use, we'll also need to backfill the new cluster with old data. This could be implemented as a background migration that performs a staggered SCAN across the keyspace with special casing on redis key types. If this is potentially expensive, we'll want a way to turn it off (so that it doesn't run during peak).

Edited by Igor