Add a timeout to rules evaluation in pipeline creation

What does this MR do and why?

Pipeline creation evaluates rules for every job defined in the config, including jobs that rules exclude. A config with thousands of such jobs can keep pipeline creation busy for many minutes and use a lot of CPU and memory.

This MR adds a deadline to the seed step, similar to the include fetch timeout (GITLAB_CI_CONFIG_FETCH_TIMEOUT_SECONDS):

  • The deadline is checked before the rules of each job candidate are evaluated (Seed::Build#included?).
  • When it expires, the pipeline fails with the new seed_timeout failure reason and this error: Pipeline creation timed out while evaluating rules. Pipelines with many jobs defined in the configuration can take longer to evaluate.
  • The error log includes job_candidates_count and evaluated_job_candidates_count, to help size a future limit on job candidates.
  • The default timeout is 900 seconds. Self-managed admins can change it with GITLAB_CI_PIPELINE_SEED_TIMEOUT_SECONDS (1 to 1800).

Feature flags:

  • ci_pipeline_seed_timeout (beta, actor: project) enables the timeout. Off by default.
  • ci_pipeline_seed_timeout_override (ops, actor: root namespace) raises the timeout to 1800 seconds, like ci_config_fetch_timeout_override does for the include fetch timeout.

References

Screenshots or screen recordings

Not applicable, no UI changes.

How to set up and validate locally

  1. Set GITLAB_CI_PIPELINE_SEED_TIMEOUT_SECONDS=1 in env.runit and restart GDK (gdk restart rails-background-jobs rails-web).
  2. Enable the feature flag:
    Feature.enable(:ci_pipeline_seed_timeout, Project.find_by_full_path('<project>'))
  3. Push a .gitlab-ci.yml with many jobs that rules exclude, for example a large parallel:matrix with rules: [{ if: $CI_COMMIT_TAG }].
  4. The pipeline fails with the seed_timeout failure reason and the error above.
  5. Enable ci_pipeline_seed_timeout_override for the root namespace, or disable ci_pipeline_seed_timeout, and push again. The pipeline is created.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Furkan Ayhan

Merge request reports

Loading
Loading