Recover and revert stuck-stopping environments

Why

See Environments are stuck in `stopping` state if t... (#363197 - closed) for full context.

In summary, when on stopping job fails, environments can get stuck in a state that makes it difficult for users to work with. We want to improve the experience by ensuring environment states are predictable and easy to work with.

What

  • introduce logic (maybe a service, maybe something simpler) to recover a stuck stopping job
  • implement logic to always recover a stopping job that fails
  • keep in mind the code to recover should be put in a place that will be reusable in a cron job as a 2nd iteration
Edited by Hunter Stewart