Keep job available after failure for a period of time
Description
As discussed in https://gitlab.com/gitlab-org/gitlab-ce/issues/25990#note_60994065 the user should be able to access the terminal after the job has failed for a specific period of time. This will require the runner to keep the the pods/containers available in case the user wants to accesss the terminal to debug an issue.
Proposal
- Manage life cycle of pods/containers if the job is a failure to keep it around (period of time configurable)
- Have a clean up job to delete pods/containers
Links to related issues and merge requests / references
gitlab-org/gitlab-ce#25990