DRI @msmiley ### Background At the end of &469, the project to have a single queue per shard, we will also be turning off all VMs that were used for Sidekiq. Sidekiq will run exclusively on Kubernetes. It is possible that we aren't running Sidekiq in "the Kubernetes way" and we need to do some research on the best strategy for this new environment. In the VM environment, workers could specify their properties so that they could be correctly routed to shards that were provisioned to meet their requirements. These are `memory-bound, cpu-bound` etc. In the Kubernetes environment, there are other bounded resources that could potentially be tuned to better service the needs of the workers. And if the workers could specify their needs of the bounded resources, this could be automated. ### Proposal Engage with Development Engineering to perform an analysis of all workers. Using data over a specified time frame, we could build a set of resource requirements for each worker. The engagement helps the stage groups increase their knowledge of how their workers consume resources, and to carefully consider what happens if the upper bound of the resource is exceeded. We can analyze this data to: - see how to provision the Kubernetes instances, and - engage with teams to address resource consumption --- Rough ideas: * separate HPAs for each job class * sidekiq job concurrency per pod is too high * maybe ask job classes to specify their bounding resource in some way other than just categorical?