Loading
Add a timeout to rules evaluation in pipeline creation
What does this MR do and why?
Pipeline creation evaluates rules for every job defined in the config, including jobs that rules exclude. A config with thousands of such jobs can keep pipeline creation busy for many minutes and use a lot of CPU and memory.
This MR adds a deadline to the seed step, similar to the include fetch timeout (GITLAB_CI_CONFIG_FETCH_TIMEOUT_SECONDS):
- The deadline is checked before the rules of each job candidate are evaluated (
Seed::Build#included?). - When it expires, the pipeline fails with the new
seed_timeoutfailure reason and this error:Pipeline creation timed out while evaluating rules. Pipelines with many jobs defined in the configuration can take longer to evaluate. - The error log includes
job_candidates_countandevaluated_job_candidates_count, to help size a future limit on job candidates. - The default timeout is 900 seconds. Self-managed admins can change it with
GITLAB_CI_PIPELINE_SEED_TIMEOUT_SECONDS(1 to 1800).
Feature flags:
ci_pipeline_seed_timeout(beta, actor: project) enables the timeout. Off by default.ci_pipeline_seed_timeout_override(ops, actor: root namespace) raises the timeout to 1800 seconds, likeci_config_fetch_timeout_overridedoes for the include fetch timeout.
References
Screenshots or screen recordings
Not applicable, no UI changes.
How to set up and validate locally
- Set
GITLAB_CI_PIPELINE_SEED_TIMEOUT_SECONDS=1inenv.runitand restart GDK (gdk restart rails-background-jobs rails-web). - Enable the feature flag:
Feature.enable(:ci_pipeline_seed_timeout, Project.find_by_full_path('<project>')) - Push a
.gitlab-ci.ymlwith many jobs thatrulesexclude, for example a largeparallel:matrixwithrules: [{ if: $CI_COMMIT_TAG }]. - The pipeline fails with the
seed_timeoutfailure reason and the error above. - Enable
ci_pipeline_seed_timeout_overridefor the root namespace, or disableci_pipeline_seed_timeout, and push again. The pipeline is created.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Edited by Furkan Ayhan