Use multiple Redis cache instances in Rails.cache
Requires: gitlab-org/gitlab#30234 (closed)
Impacted Service: redis-cache
Impacted Saturation Point: single_threaded_cpu
GitLab will soon be running on Rails 5.2: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/12145
Rails 5.2 supports an option of running multiple Redis independent instances as the caching backend.
This approach has the advantage of being operationally easy to run, compared to Redis Cluster.
Since our Redis cache workload is far-and-away read orientated, this approach might work well.
More details of multiple Redis stores can be found here: https://www.engineyard.com/blog/rails-5-2-redis-cache-store
Multiple Redis Servers
You can use multiple Redis servers for sharding. If one Redis server goes down, you'll have cache misses for keys that are stored on that server, but the other keys on the remaining servers would still work. Writes to the unavailable server would be dropped. There would be no exceptions raised and your app will continue running.
To specify multiple servers, pass the url option.
redis_servers = %w[ redis://hostname-1:6379/0 redis://hostname-2:6379/0 redis://hostname-3:6379/0 redis://hostname-4:6379/0 ] config.cache_store = :redis_cache_store, {driver: :hiredis, url: redis_servers}
By specifying multiple servers, Redis::Distributed is used, which is part of the redis gem.
This could be an alternative to gitlab-org/gitlab#29887 (closed), or possibly used in parallel.
corrective action for incidents including