Performance Indicator - Sidekiq

As discussed here: #108 (closed)

We would like to have a set of performance indicators that help the team prioritize future work. Since it is difficult to list all services and define all indicators in one go, we are going to do this iteratively as we move through the system. We have recently completed work on Sidekiq and we can take these learnings to define a performance indicator based on headroom for Sidekiq.

Under and Over Provisioning

While discussing what would be an appropriate PI for sidekiq, we realised that under-provisioning and over-provisioning should be viewed as two separate indicators.

Under-Provisioning

Not having enough resources to meet demand is more likely to result in customer impact. Job urgency targets would be missed and our service level objectives would be at risk. These two indicators show us when we do not have enough resources to service jobs.

We need to find out how current alerting around under-provisioning works and see how we could reuse this in a performance indicator.

Over-Provisioning

Having too many resources that are idle most of the time is not an efficient use of resources and is a waste of money. Currently, it is an ad-hoc manual process to identify Sidekiq shards that are over-provisioned. A first iteration might be to have a person perform this analysis on a schedule, but it would be more efficient to have an automated way to be alerted to this because it is an unclear process to repeat and relies on a lot of personal judgment.