Skip to content

Possible application-side performance regressions in cache due to Redis Cluster

When using Redis Cluster, mget are converted into pipelines of get. While this lets us scale redis-cache horizontally, it could lead to a slowdown on the application-side for requests with very big multi-key commands

From logs (https://log.gprd.gitlab.net/app/r/s/lR73X), we can pick out a few candidates for optimisation:

  • Projects::CompareController#show makes a pipeline of hmget even for a standard Redis. In redis-rb v5, the multiple pipeline calls will be made concurrently via threads (performance impact to be determined)
[lib/gitlab/instrumentation/redis_interceptor.rb:26:in `read',
 lib/gitlab/instrumentation/redis_interceptor.rb:16:in `block in call_pipeline',
 lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call',
 lib/gitlab/instrumentation/redis_interceptor.rb:15:in `call_pipeline',
 lib/gitlab/redis/cross_slot.rb:110:in `block in execute_commands',
 lib/gitlab/redis/cross_slot.rb:108:in `each',
 lib/gitlab/redis/cross_slot.rb:108:in `execute_commands',
 lib/gitlab/redis/cross_slot.rb:94:in `pipelined',
 lib/gitlab/markdown_cache/redis/store.rb:14:in `block (2 levels) in bulk_read',
 lib/gitlab/instrumentation/redis_cluster_validator.rb:208:in `allow_cross_slot_commands',
 lib/gitlab/markdown_cache/redis/store.rb:13:in `block in bulk_read',
  • GET /api/:version/projects uses a Rails.cache where mget commands are now converted into pipelines of get. We could increase the pipeline batch size to reduce the number of round-trips with the Redis Cluster nodes
[lib/gitlab/instrumentation/redis_interceptor.rb:26:in `read',
 lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call',
 lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call',
 lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call',
 lib/gitlab/cache/helpers.rb:120:in `block in fetch_multi',
 lib/gitlab/instrumentation/redis_cluster_validator.rb:208:in `allow_cross_slot_commands',
 lib/gitlab/cache/helpers.rb:119:in `fetch_multi',
 lib/gitlab/cache/helpers.rb:93:in `cached_collection',
 lib/api/helpers/caching.rb:48:in `present_cached',

Note: WIP, to add graphs on multi-key command sizes

Edited by Sylvester Chin