## Summary Roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/230249) currently behind the `ci_stick_build_after_commit` feature flag. - DRI: @hfyngvason - Team Slack channel: `#g_pipeline-execution` > [!note] > Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? The feature changes when running CI builds are stuck to the primary database. If the post-commit timing is incorrect, runners or code paths that rely on immediate sticking after a build transitions to `running` could observe stale replica data. Watch CI build transition and runner request behavior during rollout. If stale-read symptoms or job-pickup regressions appear, roll back by disabling the feature flag. See https://dashboards.gitlab.net/goto/efrjbowsjwjy8d?orgId=1 (internal link) for server-side timeout of jobs stuck running, query: ```promql sum( increase(gitlab_ci_job_failure_reasons{reason="server_timeout_running"}[$__rate_interval]) ) ``` See also the [the pending -> running mismatch logs](https://log.gprd.gitlab.net/app/r/s/i6du8). The latter is a known cause of the former. - **Expected results:** We are hoping for both graphs to go down. - **Rollback signal:** Graphs go up instead. - **Neutral:** If we see no effect, we should likely still keep the fix (because it does fix a theoretical bug), but we won't be able to close out the bug issue. Finally, see - [the transitions to running](https://dashboards.gitlab.net/goto/ffrkz4comark0e?orgId=default) for anomaly detection (in case there are errors) - [related APIs for anomalies](https://dashboards.gitlab.net/d/general-rails-controller/general3a-rails-controller?orgId=1&from=now-6h%2Fm&to=now%2Fm&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-type=api&var-stage=main&var-controller=Grape&var-action=POST%20%2Fapi%2Fjobs%2Frequest&var-action=PUT%20%2Fapi%2Fjobs%2F:id&var-action=PATCH%20%2Fapi%2Fjobs%2F:id%2Ftrace), e.g. increased error rate, database anomalies, performance anomalies etc ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#g_pipeline-execution`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set ci_stick_build_after_commit 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set ci_stick_build_after_commit true --dev --pre --staging --staging-ref ``` **Production** — percentage rollout (wait ≥15 min between steps, watch dashboards): ``` /chatops gitlab run feature set ci_stick_build_after_commit <percentage> --actors ``` Or target specific actors instead: ``` /chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss ci_stick_build_after_commit true /chatops gitlab run feature set --group=gitlab-org,gitlab-com ci_stick_build_after_commit true /chatops gitlab run feature set --user=hfyngvason ci_stick_build_after_commit true ``` ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [ ] [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [ ] [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [ ] [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [ ] [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check https://gitlab.com/gitlab-org/gitlab/-/merge_requests/230249 19.2 /chatops gitlab run feature delete ci_stick_build_after_commit --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set ci_stick_build_after_commit false # production /chatops gitlab run feature set ci_stick_build_after_commit false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete ci_stick_build_after_commit --dev --pre --staging --staging-ref --production # remove entirely ```