[Controller] Deploy Webservice, Sidekiq, and Task Runner after migrations are done
Summary
Currently, the controller deploys in this order:
- Deploy all StatefulSets
- Wait for all StatefulSets to be completely up
- Deploy all Deployments
This means that the Migrations pod is running while Webservice, Sidekiq, and Task Runner are attempting to start. They will be in CrashLoopBackOff until migrations are done, which can lead to a longer end-to-end release because the CrashLoopBackOff period gets longer with each crash.
As a workaround, we can update the controller to only deploy those 3 Deployments after Migrations pod has completed.
Acceptance criteria
-
Webservice, Sidekiq, and Task Runner (to be Toolbox) Deployments are only submitted after Migrations are complete -
No CrashLoopBackOff is observed in the above Deployments due to incomplete migrations
Edited by Mitchell Nielsen