Define an SLO for rails pod startup times
There have been several discussions around pod startup time:
We should define an SLA for the time it takes for a pod to become ready.
Having an SLA for this will help with setting up an efficient autoscaling strategy. Additionally, if performance regressions lead to the application no longer meeting the SLA, we can alert on this and investigate.
Example SLA: 95% of web, api, git and sidekiq pods should start with 1 minute.
Measuring startup durations
Some investigation into the best approach to measuring this value would be needed, but it's likely we could use kube-state-metrics to determine the startup time for pods. https://github.com/kubernetes/kube-state-metrics