Add runtime_environment_unavailable failure reason and keep environment keys past the runner TTL

Ashvin Sharmarequested to merge
ashvins/runtime-environment-ttl into master

What does this MR do and why?

GitLab Runner now enforces a TTL on suspended environments (default 7 days, max 30). When a job resumes an environment that has expired, was released, or was lost, the runner fails it with runtime_environment_unavailable. This MR makes Rails support that, as the Runner TTL design requires.

  • Adds failure reason runtime_environment_unavailable (enum 34). It is advertised to the runner automatically through features.failure_reasons, and gets a presenter message, a status description, GraphQL docs, and a gitlab.pot entry.
  • Never retries it automatically (RETRY_OVERRIDES: 0), and marks it unrecoverable: the environment is gone, and a retried job loses its resume key. Following the precedent of environment_creation_failure, it is not listed in the retry:when docs or schema.
  • Raises Ci::RuntimeEnvironment::PARTITION_CLEANUP_THRESHOLD from 7 to 31 days. Partitions were detached after 7 days, so a resume after day 7 got no key and silently ran on a fresh environment. The threshold is now longer than the runner's 30-day maximum TTL, as the design requires. This is a constant read by the partition manager, so there is no schema change and no migration.

References

How to set up and validate locally

  1. bin/rspec spec/lib/gitlab/ci/build/auto_retry_spec.rb spec/models/commit_status_spec.rb spec/requests/api/ci/runner/jobs_put_spec.rb spec/models/ci/runtime_environment_spec.rb spec/presenters/commit_status_presenter_spec.rb spec/lib/gitlab/ci/status/build/failed_spec.rb spec/lib/gitlab/ci/config/entry/retry_spec.rb
  2. A runner PUT /jobs/:id with failure_reason=runtime_environment_unavailable stores that reason, and the job is not auto-retried even with retry: 2.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Ashvin Sharma

Merge request reports