## Summary Roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/work_items/635122) currently behind the `ci_pipeline_seed_timeout` feature flag. - DRI: @furkanayhan - Team Slack channel: `#g_pipeline-authoring` > [!note] > Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? - Pipelines that take longer than the seed timeout (900 seconds by default) to evaluate `rules` start failing with the `seed_timeout` failure reason. - To give a namespace more time, enable `ci_pipeline_seed_timeout_override` for its root namespace. This raises the timeout to 1800 seconds. - Watch `gitlab_ci_pipeline_failure_reasons{reason="seed_timeout"}` and the `Gitlab::Ci::Pipeline::Seed::Context::TimeoutError` logs (`job_candidates_count`, `evaluated_job_candidates_count`). ## Monitoring **Kibana** - [Events with `ci_pipeline_seed_timeout:1`](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.feature_flag_states:%22ci_pipeline_seed_timeout:1%22'),columns:!(json.meta.caller_id,json.feature_flag_states),sort:!(!(json.time,desc)))) (Rails only) - Timeout errors (`Gitlab::Ci::Pipeline::Seed::Context::TimeoutError`): [Sidekiq](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'AWNABDRwNDuQHTm2tH6l',query:(language:kuery,query:'json.exception.class.keyword:%22Gitlab::Ci::Pipeline::Seed::Context::TimeoutError%22'),columns:!(json.meta.project,json.meta.root_namespace,json.extra.pipeline_source,json.extra.job_candidates_count,json.extra.evaluated_job_candidates_count,json.meta.caller_id),sort:!(!(json.time,desc)))) / [Rails](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.exception.class.keyword:%22Gitlab::Ci::Pipeline::Seed::Context::TimeoutError%22'),columns:!(json.meta.project,json.meta.root_namespace,json.extra.pipeline_source,json.extra.job_candidates_count,json.extra.evaluated_job_candidates_count,json.meta.caller_id),sort:!(!(json.time,desc)))) - Other exceptions with `ci_pipeline_seed_timeout:1` (not the timeout itself): [Sidekiq](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'AWNABDRwNDuQHTm2tH6l',query:(language:kuery,query:'json.exception.feature_flag_states.keyword:%22ci_pipeline_seed_timeout:1%22%20and%20not%20json.exception.class.keyword:%22Gitlab::Ci::Pipeline::Seed::Context::TimeoutError%22'),columns:!(json.exception.class,json.exception.message,json.meta.project,json.meta.caller_id),sort:!(!(json.time,desc)))) / [Rails](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.exception.feature_flag_states.keyword:%22ci_pipeline_seed_timeout:1%22%20and%20not%20json.exception.class.keyword:%22Gitlab::Ci::Pipeline::Seed::Context::TimeoutError%22'),columns:!(json.exception.class,json.exception.message,json.meta.project,json.meta.caller_id),sort:!(!(json.time,desc)))) - Slow seed step, ≥600s (pipelines close to the 900s timeout): [Sidekiq](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'AWNABDRwNDuQHTm2tH6l',query:(language:kuery,query:'json.pipeline_step_gitlab_ci_pipeline_chain_seed_duration_s%20%3E=%20600'),columns:!(json.meta.project,json.meta.root_namespace,json.pipeline_source,json.pipeline_persisted,json.pipeline_step_gitlab_ci_pipeline_chain_seed_duration_s,json.pipeline_seed_build_inclusion_duration_s.count,json.pipeline_seed_build_inclusion_duration_s.sum),sort:!(!(json.time,desc)))) / [Rails](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.pipeline_step_gitlab_ci_pipeline_chain_seed_duration_s%20%3E=%20600'),columns:!(json.meta.project,json.meta.root_namespace,json.pipeline_source,json.pipeline_persisted,json.pipeline_step_gitlab_ci_pipeline_chain_seed_duration_s,json.pipeline_seed_build_inclusion_duration_s.count,json.pipeline_seed_build_inclusion_duration_s.sum),sort:!(!(json.time,desc)))) **Grafana** - [Pipeline creation failure ratio (CI Orchestration: Pipeline Observability)](https://dashboards.gitlab.net/d/ci-orchestration-pipeline-observability?viewPanel=panel-25&var-environment=gprd&var-stage=main&from=now-24h&to=now) - [`gitlab_ci_pipeline_failure_reasons{reason="seed_timeout"}` in Explore](https://dashboards.gitlab.net/explore?schemaVersion=1&orgId=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22mimir-gitlab-gprd%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22sum%20by%20%28reason%29%20%28rate%28gitlab_ci_pipeline_failure_reasons%7Benv%3D%5C%22gprd%5C%22%2Creason%3D%5C%22seed_timeout%5C%22%7D%5B5m%5D%29%29%22%2C%22range%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22mimir-gitlab-gprd%22%7D%2C%22editorMode%22%3A%22code%22%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-24h%22%2C%22to%22%3A%22now%22%7D%7D%7D) **Sentry** - [Errors in `lib/gitlab/ci/pipeline/seed/context.rb`](https://new-sentry.gitlab.net/organizations/gitlab/issues/?project=3&query=is%3Aunresolved%20stack.filename%3A%22lib%2Fgitlab%2Fci%2Fpipeline%2Fseed%2Fcontext.rb%22&statsPeriod=24h) - [Errors in `lib/gitlab/ci/pipeline/chain/seed.rb`](https://new-sentry.gitlab.net/organizations/gitlab/issues/?project=3&query=is%3Aunresolved%20stack.filename%3A%22lib%2Fgitlab%2Fci%2Fpipeline%2Fchain%2Fseed.rb%22&statsPeriod=24h) - [Errors in `lib/gitlab/ci/pipeline/seed/build.rb`](https://new-sentry.gitlab.net/organizations/gitlab/issues/?project=3&query=is%3Aunresolved%20stack.filename%3A%22lib%2Fgitlab%2Fci%2Fpipeline%2Fseed%2Fbuild.rb%22&statsPeriod=24h) ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#g_pipeline-authoring`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set ci_pipeline_seed_timeout 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set ci_pipeline_seed_timeout true --dev --pre --staging --staging-ref ``` **Production** — percentage rollout (wait ≥15 min between steps, watch dashboards): ``` /chatops gitlab run feature set ci_pipeline_seed_timeout <percentage> --actors ``` Or target specific actors instead: ``` /chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss ci_pipeline_seed_timeout true /chatops gitlab run feature set --group=gitlab-org,gitlab-com ci_pipeline_seed_timeout true /chatops gitlab run feature set --user=furkanayhan ci_pipeline_seed_timeout true ``` ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/work_items/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete ci_pipeline_seed_timeout --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set ci_pipeline_seed_timeout false # production /chatops gitlab run feature set ci_pipeline_seed_timeout false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete ci_pipeline_seed_timeout --dev --pre --staging --staging-ref --production # remove entirely ```