Enforce prepare_timeout config field

Daniel Keenanrequested to merge
implement-prepare-stage-timeout into main

What does this MR do?

  • Cancels a job if the prepare stage (the prepare_executor and prepare_script stages combined) takes longer than the configured prepare_timeout.
  • Surfaces the timeout as ERROR: Job failed (system failure): prepare timeout in the job output.
  • Subsequent stages (e.g., get_sources) are NOT bounded by prepare_timeout.

Why was this MR needed?

Solves the problem of jobs hanging for a long time (up to the full job timeout) and occupying concurrency slots when the prepare stage is taking too long (e.g., due to a slow/hung image pull).

Part of #26583 (closed)

What's the best way to test this MR?

Timeout Path:

  1. Configure a runner with a very short prepare_timeout (e.g., 1ms) in your config.toml
  2. Execute a job using that runner
  3. Verify the job fails with ERROR: Job failed (system failure): prepare timeout

Happy Path:

  1. Configure a runner with a reasonable prepare_timeout (e.g., 30s) in your config.toml
  2. Execute a job using that runner
  3. Verify the job completes successfully

What are the relevant issue numbers?

Implements #39373 (closed), #39374 (closed)

Merge request reports