## Summary Roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/work_items/602624) currently behind the `ci_pipeline_mr_main_db_wal_pinning` feature flag. The flag pins the main database WAL location for merge request pipelines at pipeline creation, and has `Ci::RegisterJobService` ensure the replica it reads from is at least as up-to-date before serializing the job. This guarantees the runner sees the same or newer merge request state as the pipeline creation logic (important for newly created MRs). The change is diagnostic in part: it helps eliminate replication lag as a factor while we confirm the true cause. - DRI: @hfyngvason - Team Slack channel: `#g_pipeline-execution` - Introducing MR: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/240965 - Customer issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/602624 > [!note] > Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? Runners may receive HTTP 409 and retry more often when the main replica is lagging, slightly increasing register-job retries. Some jobs will stay in `pending` for longer than before, but only jobs that would otherwise have failed. No data loss expected. Monitor to detect problems: - **Expected to drop:** the `git_issues_remote_ref_error` failure signature on the [DX Failure Analysis dashboard](https://dashboards.gitlab.net/d/dx-failure-analysis/dx3a-failure-analysis-dashboard?from=now-30d&to=now&timezone=utc&var-project_path_var=$__all&var-failure_category_var=git_issues_remote_ref_error&var-failure_signature_var=git_issues_remote_ref_error__36330747c724a643&orgId=1) (signature `36330747c724a643`). - [`:queue_merge_request_replication_lag` queue operation counter](https://dashboards.gitlab.net/goto/dfq7gux5dzdhcc?orgId=1) in `Ci::RegisterJobService`. - Register-job retry/conflict rates. ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#g_pipeline-execution`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set ci_pipeline_mr_main_db_wal_pinning 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set ci_pipeline_mr_main_db_wal_pinning true --dev --pre --staging --staging-ref ``` **Production** — percentage rollout (wait ≥15 min between steps, watch dashboards): ``` /chatops gitlab run feature set ci_pipeline_mr_main_db_wal_pinning <percentage> --actors ``` Or target specific actors instead: ``` /chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss ci_pipeline_mr_main_db_wal_pinning true /chatops gitlab run feature set --group=gitlab-org,gitlab-com ci_pipeline_mr_main_db_wal_pinning true /chatops gitlab run feature set --user=hfyngvason ci_pipeline_mr_main_db_wal_pinning true ``` ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete ci_pipeline_mr_main_db_wal_pinning --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set ci_pipeline_mr_main_db_wal_pinning false # production /chatops gitlab run feature set ci_pipeline_mr_main_db_wal_pinning false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete ci_pipeline_mr_main_db_wal_pinning --dev --pre --staging --staging-ref --production # remove entirely ```