Improve reconcile loop performance by running early checks
For certain checks, e.g. checking readiness SideKiq, WebService, of other core services, we use re-queue and retry the check after a wait period.
This is a good approach. However, the way that it has been implemented is problematic. For example, the status check is done at the very end of a long-running loop. There is a good chance that the loop is started because the original request was re-queued. Statistically the reconcile loop is called more often as a result of a re-queue rather than an original request.
It is a good idea to run these checks very early, esp. before rendering the template (which is time-consuming and takes few seconds) and break as early as possible.