Split Redis-Sidekiq from Redis-Persistent

Currently at GitLab we have two Redis clusters in production

  1. Redis Persistent: this uses frequent RDB snapshotting to achieve (relatively high) persistence in our Redis cluster. It is used for everything that is not cache data. The three main uses are

    1. Sidekiq/Resque Queues: Sidekiq operates on this instance. All jobs are pushed into Redis and workers pop jobs from Redis lists to obtain jobs.
    2. Session State: we use Rails Redis Session Store, with extensions to allow the the enumeration of the sessions for a user. During DDOS attacks, this can lead to pressure on Redis Persistent.
    3. Etag Support: this is an efficient caching mechanism
    4. Workhorse Long-Polling Coordination: Workhorse and Rails coordinate long-polling through Redis.

  2. Redis Cache: the Redis cache contains all cache information. This is primarily accessed through the Rails.cache interface.

Proposal: split Redis Sidekiq from Redis Persistent

Why?

  • Sidekiq queueing and the other concerns are not related. Issues with Sidekiq should not affect, for example Session State, and DDOS attacks should not affect our worker queues.
  • Further splits for other components can be made in future. This is the first step, but a logical first step.
  • Workloads for Sidekiq workers tend to be quite different from other Redis clients - lots of blocking operations.

Migration

Migration can be done without downtime.

  1. Make application changes & omnibus changes to allow Sidekiq to use a different Redis than the normal Redis persistent.
  2. Stand up a new Redis Primary/Secondary/Sentinel fleet (better yet, reuse the existing Sentinels. Sentinel is designed to monitor multiple clusters)
  3. Stand up a new Sidekiq fleet, pointing it at the new Redis-Sidekiq. It will initially be idle.
  4. Update the application (Web, API, etc) to send jobs to the new Redis-Sidekiq.
  5. The new Sidekiq fleet will now start processing jobs as they arrive.
  6. Wait for the old Sidekiq fleet to finish all it's pending jobs (leave it for a few hours)
  7. Decommission the old Sidekiq fleet.

Why not use Redis Cluster?

Redis Cluster is not intended for workloads like Sidekiq. In the words of the Sidekiq author:

"Cluster is for caching and similar workloads that can scale horizontally. That is not Sidekiq."

https://github.com/mperham/sidekiq/issues/4012

Additionally, I don't believe that we have the expertise to run a production Redis Cluster. Cluster is much more complicated to run than non-clustered Redis. Let's focus on boring solutions.

cc @glopezfernandez @dawsmith @Finotto @ansdval

Related production issue: production#963 (closed)

Edited by Gerardo Lopez-Fernandez