Proposal: Scaling Sidekiq with Zonal Clusters
The Redis-Sidekiq service is one of the more concerning services from a saturation point of view.
There have been various discussions around how to deal with this, including:
Additionally, the Redis 6 upgrade, with io-threads
should provide some relief.
Ultimately, we will need to scale this fleet horizontally.
Proposal Zonal Sidekiq Clusters
- We setup a Redis-Sidekiq (+sentinel, etc) replica-set in each Kubernetes zonal fleet
- We setup a Sidekiq worker fleet inside each zonal cluster
- We configure all Rails services, include Sidekiq itself to use the local Sidekiq/Redis-Sidekiq instance
- Once a Sidekiq job is issued within a particular cluster, and further upstream jobs will remain in the same cluster
- We monitor using the new Regional SLO monitoring
This approach has several advantages and some downsides too, but one of the main reasons I really like it is because of the simplicity of the approach.
@jarv also points out that having Sidekiq running inside the regional clusters will allow us to quickly halt the deployment on Sidekiq issues, something that we are currently unable to do. cc @amyphillips
![](/-/project/14231319/uploads/15b956df301207fd7ea3b7cafdfaa3a2/image.png)
More details to follow
Edited by Rachel Nienaber