Recover and long running stuck-stopping environments
Why
See Environments are stuck in `stopping` state if t... (#363197 - closed) for full context. This issue depends on Recover and revert stuck-stopping environments (#425161 - closed) (first iteration of work) to provide the logic to recover.
In summary, when on stopping job fails, environments can get stuck in a state that makes it difficult for users to work with. We want to improve the experience by ensuring environment states are predictable and easy to work with.
What
Introduce a cron job to revert the states if environments are stuck for more than an reasonable period. StuckCiJobsWorker is a good example of this.
- add new job similar to StuckCiJobsWorker
- reuse logic introduced in Recover and revert stuck-stopping environments (#425161 - closed) to recover a stuck stopping job