Possible application-side performance regressions in cache due to Redis Cluster
When using Redis Cluster, mget
are converted into pipelines of get
. While this lets us scale redis-cache
horizontally, it could lead to a slowdown on the application-side for requests with very big multi-key commands
From logs (https://log.gprd.gitlab.net/app/r/s/lR73X), we can pick out a few candidates for optimisation:
-
Projects::CompareController#show
makes a pipeline ofhmget
even for a standard Redis. In redis-rb v5, the multiple pipeline calls will be made concurrently via threads (performance impact to be determined)
[lib/gitlab/instrumentation/redis_interceptor.rb:26:in `read',
lib/gitlab/instrumentation/redis_interceptor.rb:16:in `block in call_pipeline',
lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call',
lib/gitlab/instrumentation/redis_interceptor.rb:15:in `call_pipeline',
lib/gitlab/redis/cross_slot.rb:110:in `block in execute_commands',
lib/gitlab/redis/cross_slot.rb:108:in `each',
lib/gitlab/redis/cross_slot.rb:108:in `execute_commands',
lib/gitlab/redis/cross_slot.rb:94:in `pipelined',
lib/gitlab/markdown_cache/redis/store.rb:14:in `block (2 levels) in bulk_read',
lib/gitlab/instrumentation/redis_cluster_validator.rb:208:in `allow_cross_slot_commands',
lib/gitlab/markdown_cache/redis/store.rb:13:in `block in bulk_read',
-
GET /api/:version/projects
uses aRails.cache
wheremget
commands are now converted into pipelines ofget
. We could increase the pipeline batch size to reduce the number of round-trips with the Redis Cluster nodes
[lib/gitlab/instrumentation/redis_interceptor.rb:26:in `read',
lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call',
lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call',
lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call',
lib/gitlab/cache/helpers.rb:120:in `block in fetch_multi',
lib/gitlab/instrumentation/redis_cluster_validator.rb:208:in `allow_cross_slot_commands',
lib/gitlab/cache/helpers.rb:119:in `fetch_multi',
lib/gitlab/cache/helpers.rb:93:in `cached_collection',
lib/api/helpers/caching.rb:48:in `present_cached',
Note: WIP, to add graphs on multi-key command sizes
Edited by Sylvester Chin