Spin up a low-urgency-cpu-bound shard for sidekiq
As a PoC, we will spin up a low-urgency-cpu-bound
shard for processing jobs, first in staging then in production.
We can close this issue when both staging an production are running a shard that's processing jobs next to the regular priority based nodes.
When both are running and processing the expected jobs, we can proceed to spin up the other shards in #27 (closed).
We're spinning up that shard next to our existing sidekiq priorities.
The rollout plan could then be as follows:
- Start a single VM with a selector running sidekiq cluster and watch it process jobs
- Start VMs for all queries we intend to end up with and watch them process jobs
- Scale up number of VMs using the selectors to the number we expect we need.
- Scale down the old "priority" based VMs.
Edited by Bob Van Landuyt