## Summary Roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/work_items/591467) currently behind the `ci_register_job_phase_timeouts` feature flag. - DRI: @hfyngvason - Team Slack channel: `#g_pipeline-execution` > [!note] > Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? This flag adds a single internal time budget to `Ci::RegisterJobService`: 55 seconds, measured from when the service starts handling the request. The pre-assign runner checks and the rendering of the runner response each run against whatever remains of that budget when they start, rather than getting a fixed allowance. Builds whose combined pre-assign checks and response rendering can't fit in the remaining time will now be dropped immediately with `scheduler_failure`, instead of eventually succeeding (or failing much later via the stuck-build sweep or project timeout). This is the intended trade-off: those builds were mostly dying anyway once they hit the request's overall 60-second `Rack::Timeout`, just later and less cleanly. The risk is that the budget is sized too tightly and starts failing builds that previously succeeded, just slowly. ### Monitoring Baselines below are from 2026-08-28, with the flag off. Watch the new timeout counters and `build_conflict_exception` first — they show directly whether the flag is dropping builds. One risk is not covered elsewhere in this issue: `with_phase_timeout` uses `Timeout.timeout`, which can interrupt a thread mid-read on a PG socket. If that happens, the connection is left with an unread result, the next query on it fails, and the load balancer marks a healthy replica offline. The same interrupt risk applies to the Gitaly gRPC calls in `present_build!` and to Redis. #### Grafana | Link | What it shows | Baseline (flag off) | | --- | --- | --- | | [G1 — New timeout counters (the flag's own signal)](<https://dashboards.gitlab.net/explore?orgId=1&schemaVersion=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22mimir-gitlab-gprd%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22sum%28rate%28gitlab_ci_queue_operations_total%7Benv%3D%5C%22gprd%5C%22%2Ctype%3D%5C%22api%5C%22%2Coperation%3D%5C%22queue_phase_timeout_pre_assign_runner_checks%5C%22%7D%5B5m%5D%29%29%20or%20vector%280%29%22%2C%22range%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22mimir-gitlab-gprd%22%7D%7D%2C%7B%22refId%22%3A%22B%22%2C%22expr%22%3A%22sum%28rate%28gitlab_ci_queue_operations_total%7Benv%3D%5C%22gprd%5C%22%2Ctype%3D%5C%22api%5C%22%2Coperation%3D%5C%22queue_phase_timeout_present_build%5C%22%7D%5B5m%5D%29%29%20or%20vector%280%29%22%2C%22range%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22mimir-gitlab-gprd%22%7D%7D%2C%7B%22refId%22%3A%22C%22%2C%22expr%22%3A%22sum%28rate%28gitlab_ci_queue_operations_total%7Benv%3D%5C%22gprd%5C%22%2Ctype%3D%5C%22api%5C%22%2Coperation%3D%5C%22queue_time_budget_exhausted%5C%22%7D%5B5m%5D%29%29%20or%20vector%280%29%22%2C%22range%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22mimir-gitlab-gprd%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-6h%22%2C%22to%22%3A%22now%22%7D%7D%7D>) | Rate of `queue_phase_timeout_pre_assign_runner_checks`, `queue_phase_timeout_present_build`, and `queue_time_budget_exhausted`. | These series don't exist yet — counters are created lazily on first increment. Panels use `or vector(0)` so they don't just read "No data". | | [G2 — build_conflict_exception: rate and share of queue attempts](<https://dashboards.gitlab.net/explore?orgId=1&schemaVersion=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22mimir-gitlab-gprd%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22sum%28rate%28gitlab_ci_queue_operations_total%7Benv%3D%5C%22gprd%5C%22%2Ctype%3D%5C%22api%5C%22%2Coperation%3D%5C%22build_conflict_exception%5C%22%7D%5B5m%5D%29%29%22%2C%22range%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22mimir-gitlab-gprd%22%7D%7D%2C%7B%22refId%22%3A%22B%22%2C%22expr%22%3A%22sum%28rate%28gitlab_ci_queue_operations_total%7Benv%3D%5C%22gprd%5C%22%2Ctype%3D%5C%22api%5C%22%2Coperation%3D%5C%22build_conflict_exception%5C%22%7D%5B5m%5D%29%29%20/%20sum%28rate%28gitlab_ci_queue_operations_total%7Benv%3D%5C%22gprd%5C%22%2Ctype%3D%5C%22api%5C%22%2Coperation%3D%5C%22queue_attempt%5C%22%7D%5B5m%5D%29%29%22%2C%22range%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22mimir-gitlab-gprd%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-6h%22%2C%22to%22%3A%22now%22%7D%7D%7D>) | Rate of `build_conflict_exception`, and that rate divided by `queue_attempt`. | `build_conflict_exception` ~0/s averaged over 1h, spiking to 0.3/s. `queue_attempt` ~482/s on the api fleet. | | [G3 — Runner-visible outcome of request_job (status -1 = no response at all)](<https://dashboards.gitlab.net/explore?orgId=1&schemaVersion=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22mimir-gitlab-gprd%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22sum%20by%20%28status%29%20%28rate%28gitlab_runner_api_request_statuses_total%7Benvironment%3D%5C%22gprd%5C%22%2Cendpoint%3D%5C%22request_job%5C%22%7D%5B5m%5D%29%29%22%2C%22range%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22mimir-gitlab-gprd%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-6h%22%2C%22to%22%3A%22now%22%7D%7D%7D>) | Rate of `gitlab_runner_api_request_statuses_total` for `request_job`, broken out by status. | — | | [G4 — Endpoint SLI: apdex ratio + error rate](<https://dashboards.gitlab.net/explore?orgId=1&schemaVersion=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22mimir-gitlab-gprd%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22sum%28rate%28gitlab_sli_rails_request_apdex_success_total%7Benv%3D%5C%22gprd%5C%22%2Cendpoint_id%3D%5C%22POST%20/api/%3Aversion/jobs/request%5C%22%7D%5B5m%5D%29%29%20/%20sum%28rate%28gitlab_sli_rails_request_apdex_total%7Benv%3D%5C%22gprd%5C%22%2Cendpoint_id%3D%5C%22POST%20/api/%3Aversion/jobs/request%5C%22%7D%5B5m%5D%29%29%22%2C%22range%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22mimir-gitlab-gprd%22%7D%7D%2C%7B%22refId%22%3A%22B%22%2C%22expr%22%3A%22sum%28rate%28gitlab_sli_rails_request_error_total%7Benv%3D%5C%22gprd%5C%22%2Cendpoint_id%3D%5C%22POST%20/api/%3Aversion/jobs/request%5C%22%7D%5B5m%5D%29%29%22%2C%22range%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22mimir-gitlab-gprd%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-6h%22%2C%22to%22%3A%22now%22%7D%7D%7D>) | Apdex success ratio and error rate for `POST /api/:version/jobs/request`. | — | | [G5 — Broken DB connections on the api fleet: dead / waiting / pool saturation](<https://dashboards.gitlab.net/explore?orgId=1&schemaVersion=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22mimir-gitlab-gprd%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22sum%20by%20%28db_config_name%29%20%28gitlab_database_connection_pool_dead%7Benv%3D%5C%22gprd%5C%22%2Ctype%3D%5C%22api%5C%22%7D%29%22%2C%22range%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22mimir-gitlab-gprd%22%7D%7D%2C%7B%22refId%22%3A%22B%22%2C%22expr%22%3A%22sum%20by%20%28db_config_name%29%20%28gitlab_database_connection_pool_waiting%7Benv%3D%5C%22gprd%5C%22%2Ctype%3D%5C%22api%5C%22%7D%29%22%2C%22range%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22mimir-gitlab-gprd%22%7D%7D%2C%7B%22refId%22%3A%22C%22%2C%22expr%22%3A%22sum%28gitlab_database_connection_pool_busy%7Benv%3D%5C%22gprd%5C%22%2Ctype%3D%5C%22api%5C%22%7D%29%20/%20sum%28gitlab_database_connection_pool_size%7Benv%3D%5C%22gprd%5C%22%2Ctype%3D%5C%22api%5C%22%7D%29%22%2C%22range%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22mimir-gitlab-gprd%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-6h%22%2C%22to%22%3A%22now%22%7D%7D%7D>) | Dead and waiting connection pool gauges, plus busy/size ratio, on the api fleet. | `dead` avg 109, peak 204 over 6h. `waiting` flat at 0. | | [G6 — Replica-to-primary shift on /jobs/request (rises when replicas get marked offline)](<https://dashboards.gitlab.net/explore?orgId=1&schemaVersion=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22mimir-gitlab-gprd%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22sum%28rate%28gitlab_transaction_db_primary_count_total%7Benv%3D%5C%22gprd%5C%22%2Ctype%3D%5C%22api%5C%22%2Cendpoint_id%3D%5C%22POST%20/api/%3Aversion/jobs/request%5C%22%7D%5B5m%5D%29%29%20/%20%28sum%28rate%28gitlab_transaction_db_primary_count_total%7Benv%3D%5C%22gprd%5C%22%2Ctype%3D%5C%22api%5C%22%2Cendpoint_id%3D%5C%22POST%20/api/%3Aversion/jobs/request%5C%22%7D%5B5m%5D%29%29%20%2B%20sum%28rate%28gitlab_transaction_db_replica_count_total%7Benv%3D%5C%22gprd%5C%22%2Ctype%3D%5C%22api%5C%22%2Cendpoint_id%3D%5C%22POST%20/api/%3Aversion/jobs/request%5C%22%7D%5B5m%5D%29%29%29%22%2C%22range%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22mimir-gitlab-gprd%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-6h%22%2C%22to%22%3A%22now%22%7D%7D%7D>) | Primary reads over (primary + replica) reads for `POST /api/:version/jobs/request`. | Primary-read share 19.5% (4,626 vs 19,100 q/s). | #### Kibana | Link | What it shows | Baseline (flag off) | | --- | --- | --- | | [K1 — Phase timeout exceptions, with the build they dropped](<https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-6h,to:now))&_a=(columns:!(json.exception.class,json.extra.build_id,json.extra.project_id,json.extra.pipeline_id,json.extra.build_name),index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',interval:auto,query:(language:kuery,query:'json.subcomponent%3A%20%22exceptions_json%22%20and%20json.exception.class%3A%20%28%22Ci%3A%3ARegisterJobService%3A%3APhaseTimeoutError%22%20or%20%22Ci%3A%3ARegisterJobService%3A%3APreAssignRunnerChecksTimeoutError%22%20or%20%22Ci%3A%3ARegisterJobService%3A%3APresentBuildTimeoutError%22%29'),sort:!(!('json.time',desc)))>) | `PhaseTimeoutError`, `PreAssignRunnerChecksTimeoutError`, and `PresentBuildTimeoutError`, with build, pipeline, project id, and build name. | 0 over 24h. | | [K2 — Where the 55s budget goes: slow RegisterJobService runs, per phase](<https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-6h,to:now))&_a=(columns:!(json.total_duration_s,json.process_build_present_build_duration_s.max,json.assign_runner_failure_reason_duration_s.max,json.runner_id,json.runner_type),index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',interval:auto,query:(language:kuery,query:'json.class%3A%20%22Ci%3A%3ARegisterJobService%3A%3ALogger%22'),sort:!(!('json.time',desc)))>) | `Ci::RegisterJobService::Logger` entries with total duration and per-phase duration. | 9,416 over 24h. | | [K3 — Runs already over 45s -- the population the budget will start dropping](<https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-6h,to:now))&_a=(columns:!(json.total_duration_s,json.process_build_present_build_duration_s.max,json.assign_runner_failure_reason_duration_s.max,json.runner_id),index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',interval:auto,query:(language:kuery,query:'json.class%3A%20%22Ci%3A%3ARegisterJobService%3A%3ALogger%22%20and%20json.total_duration_s%20%3E%2045'),sort:!(!('json.time',desc)))>) | `Ci::RegisterJobService::Logger` entries where `total_duration_s > 45`. | 43 runs over 45s in 24h. | | [K4 — Load balancer connection events on the api fleet](<https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-6h,to:now))&_a=(columns:!(json.event,json.message,json.db_host,json.pod),index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',interval:auto,query:(language:kuery,query:'json.type%3A%20%22api%22%20and%20json.subcomponent%3A%20%22database_load_balancing%22%20and%20json.event%3A%20%28%22host_offline%22%20or%20%22no_secondaries_available%22%20or%20%22read_write_retry%22%20or%20%22transaction_leak%22%20or%20%22host_query_conflict%22%29'),sort:!(!('json.time',desc)))>) | `host_offline`, `no_secondaries_available`, `read_write_retry`, `transaction_leak`, and `host_query_conflict` events. | 144 total events gprd-wide over 24h: `host_offline` 24, `no_secondaries_available` 117. | | [K5 — Connection-level exceptions on the api fleet (interrupted PG connection signatures)](<https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-6h,to:now))&_a=(columns:!(json.exception.class,json.exception.message,json.pod),index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',interval:auto,query:(language:kuery,query:'json.type%3A%20%22api%22%20and%20json.subcomponent%3A%20%22exceptions_json%22%20and%20%28json.exception.class%3A%20%28%22ActiveRecord%3A%3AConnectionNotEstablished%22%20or%20%22ActiveRecord%3A%3AConnectionFailed%22%20or%20%22ActiveRecord%3A%3AStatementInvalid%22%29%20or%20json.exception.message%3A%20%28%22another%20command%20is%20already%20in%20progress%22%20or%20%22connection%20is%20closed%22%20or%20%22no%20connection%20to%20the%20server%22%29%29'),sort:!(!('json.time',desc)))>) | `ActiveRecord::ConnectionNotEstablished`, `ConnectionFailed`, `StatementInvalid`, plus messages matching interrupted-connection text. | 30 matching exceptions on api over 24h. | | [K6 — Interrupted Gitaly / Redis connections on the api fleet](<https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-6h,to:now))&_a=(columns:!(json.exception.class,json.exception.message,json.pod),index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',interval:auto,query:(language:kuery,query:'json.type%3A%20%22api%22%20and%20json.subcomponent%3A%20%22exceptions_json%22%20and%20json.exception.class%3A%20%28%22GRPC%3A%3ACancelled%22%20or%20%22GRPC%3A%3AUnavailable%22%20or%20%22GRPC%3A%3ADeadlineExceeded%22%20or%20%22Redis%3A%3AProtocolError%22%20or%20%22RedisClient%3A%3AConnectionError%22%20or%20%22Redis%3A%3ATimeoutError%22%29'),sort:!(!('json.time',desc)))>) | `GRPC::Cancelled`, `GRPC::Unavailable`, `GRPC::DeadlineExceeded`, `Redis::ProtocolError`, `RedisClient::ConnectionError`, `Redis::TimeoutError`. | 6,953 matching exceptions on api over 24h. Already noisy — watch the shape, not a threshold. | #### Sentry | Link | What it shows | Baseline (flag off) | | --- | --- | --- | | [S1 — Phase timeout errors](<https://new-sentry.gitlab.net/organizations/gitlab/issues/?project=3&statsPeriod=24h&query=error.type%3A%5B%22Ci%3A%3ARegisterJobService%3A%3APhaseTimeoutError%22%2C%22Ci%3A%3ARegisterJobService%3A%3APreAssignRunnerChecksTimeoutError%22%2C%22Ci%3A%3ARegisterJobService%3A%3APresentBuildTimeoutError%22%5D>) | Issues matching the three `PhaseTimeoutError` classes. | 0 in the last 24h. | | [S2 — Broken DB / Redis connections in CI](<https://new-sentry.gitlab.net/organizations/gitlab/issues/?project=3&statsPeriod=24h&query=feature_category%3Acontinuous_integration%20error.type%3A%5B%22ActiveRecord%3A%3AConnectionNotEstablished%22%2C%22ActiveRecord%3A%3AConnectionFailed%22%2C%22ActiveRecord%3A%3AStatementInvalid%22%2C%22PG%3A%3AUnableToSend%22%2C%22PG%3A%3AConnectionBad%22%2C%22Redis%3A%3AProtocolError%22%2C%22RedisClient%3A%3AConnectionError%22%5D>) | Issues in `feature_category:continuous_integration` matching ActiveRecord connection errors, `PG::UnableToSend`/`ConnectionBad`, and Redis/RedisClient connection errors. | 0 in the last 24h. | | [S3 — All new CI errors, sorted by first seen](<https://new-sentry.gitlab.net/organizations/gitlab/issues/?project=3&statsPeriod=24h&query=is%3Aunresolved%20feature_category%3Acontinuous_integration%20firstSeen%3A-24h>) | Unresolved `feature_category:continuous_integration` issues first seen in the last 24h. | Not a count baseline — use it to spot new error classes. `ActiveRecord::QueryCanceled` is already the largest new-CI-error class here; treat it as noise, not a signal of this flag. | One thing to keep in mind: `PG::*` never shows up directly in Kibana or Sentry, because Rails wraps those errors. K5 and S2 match on `ActiveRecord::*` classes and message text instead. ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#g_pipeline-execution`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set ci_register_job_phase_timeouts 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set ci_register_job_phase_timeouts true --dev --pre --staging --staging-ref ``` **Production** — percentage rollout (wait ≥15 min between steps, watch dashboards): ``` /chatops gitlab run feature set ci_register_job_phase_timeouts <percentage> --actors ``` Or target specific actors instead: ``` /chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss ci_register_job_phase_timeouts true /chatops gitlab run feature set --group=gitlab-org,gitlab-com ci_register_job_phase_timeouts true /chatops gitlab run feature set --user=hfyngvason ci_register_job_phase_timeouts true ``` ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/work_items/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check https://gitlab.com/gitlab-org/gitlab/-/merge_requests/250498 19.4 /chatops gitlab run feature delete ci_register_job_phase_timeouts --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set ci_register_job_phase_timeouts false # production /chatops gitlab run feature set ci_register_job_phase_timeouts false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete ci_register_job_phase_timeouts --dev --pre --staging --staging-ref --production # remove entirely ```