Proposal: Scaling Sidekiq with Zonal Clusters

The Redis-Sidekiq service is one of the more concerning services from a saturation point of view.

There have been various discussions around how to deal with this, including:

1. &194
2. #590 (closed)

Additionally, the Redis 6 upgrade, with io-threads should provide some relief.

Ultimately, we will need to scale this fleet horizontally.

Proposal Zonal Sidekiq Clusters

1. We setup a Redis-Sidekiq (+sentinel, etc) replica-set in each Kubernetes zonal fleet
2. We setup a Sidekiq worker fleet inside each zonal cluster
3. We configure all Rails services, include Sidekiq itself to use the local Sidekiq/Redis-Sidekiq instance
   1. Once a Sidekiq job is issued within a particular cluster, and further upstream jobs will remain in the same cluster
4. We monitor using the new Regional SLO monitoring

This approach has several advantages and some downsides too, but one of the main reasons I really like it is because of the simplicity of the approach.

@jarv also points out that having Sidekiq running inside the regional clusters will allow us to quickly halt the deployment on Sidekiq issues, something that we are currently unable to do. cc @amyphillips

More details to follow

cc @cmiskell @jarv @igorwwwwwwwwwwwwwwwwwwww @marin