Feature flag rollout: track_ref_pipeline_creation

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

This issue is to roll out the ref-level pipeline creation tracking that prevents auto-merge from merging before branch pipelines are created, behind the track_ref_pipeline_creation feature flag.

The fix is the reinstatement of !232296 (merged) as three merge requests: !256819 (merged) (scaffolding), !258481 (merged) (the mergeability check reads the ref-level request) and !256820 (merged) (ref-level requests created at the start of post-receive). All three are merged and deployed. The flag has not been enabled anywhere yet.

Note

Process and guidance live in the docs. This issue is just the commands and a place to track the rollout. "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability, which is not the same as releasing the feature (that happens when the flag is removed). Feature flag controls and feature flag lifecycle

What the flag does

With the flag enabled for a project, PostReceiveService registers a pipeline creation request for every pushed branch before anything else runs, and MergeRequest#pipeline_creating? reads it. While a request is in progress the CI mergeability check reports "checking", auto-merge waits, and the merge API sets auto-merge instead of merging immediately. The request is completed when the head pipeline is updated (MergeRequests::UpdateHeadPipelineWorker), when pipeline creation ends without a saved pipeline, or when CreatePipelineWorker exhausts its retries. Entries older than 300 seconds are ignored, so a lost job delays a merge by at most five minutes. Redis errors while creating requests are reported to Sentry and do not fail the push.

The actor is the project, on the push side and on the merge request's source project on the read side, so percentage rollout and --project targeting both apply per project.

What could go wrong?

  • Auto-merge delayed by up to 300 seconds when a request is left behind (lost Sidekiq job, pipeline deleted before the head pipeline update). The next push or merge request event resumes it. Watch for stuck auto-merge reports in support channels.
  • One extra Redis write per pushed branch in the internal post-receive API, and one or two extra reads in the head pipeline worker and the mergeability checks. Watch post-receive latency and Redis call counts (checks below).
  • API clients that merged immediately after a push now get auto-merge set and merge after the pipeline. That is the intended fix. Watch the merge endpoint status mix for unexpected 405/422.

Rollback is one command (below) and takes effect immediately.

Kibana checks

Before and after every production step, over equal windows. The flag's state is not logged per request (its milestone has passed), so these use routes and workers. Allow for the 15 to 35 minute ingestion lag before reading an "after" window.

pubsub-rails-inf-gprd*:

KQL Record Expect
json.route:"/api/:version/internal/post_receive" p95 json.duration_s, avg json.redis_shared_state_calls about one more Redis call per pushed branch; p95 flat
json.exception.class:* and json.meta.caller_id:"POST /api/:version/internal/post_receive" count by json.exception.class about 0; Redis* means request creation is failing
json.route:"/api/:version/projects/:id/merge_requests/:merge_request_iid/merge" count by json.status no jump in 405/422
json.exception.class:* and json.exception.message:*pipeline_creation* count by json.meta.caller_id 0

pubsub-sidekiq-inf-gprd*:

KQL Record Expect
json.class:"MergeRequests::UpdateHeadPipelineWorker" count, p95 json.duration_s, json.job_status:fail count, avg json.redis_shared_state_calls one or two more Redis calls; fails unchanged
json.class:"CreatePipelineWorker" and json.job_status:fail; json.message:"Error creating pipeline" counts unchanged; each flag-off run on the test project adds one, from the branch deleted on merge
json.class:"AutoMergeProcessWorker" count, p95 json.duration_s, fails count up slightly; duration flat
json.message:"Processing Automerge - MWCP" and json.message:"Merge request mergeable - MWCP" counts mergeable total does not fall
json.class:"Repositories::PostReceiveWorker" and json.job_status:fail count unchanged

Redis shared-state (Grafana, dashboards.gitlab.net): memory saturation, all shards, with the feature flag overlay. Expect flat: each in-progress request is one small key with a 300 second expiry, so even at 100% the live set stays in the tens of thousands of keys. A step change in saturation after a rollout step is a rollback trigger.

Rollout

Run all production /chatops in #production and cross-post to the team channel.

Pre-checks

  • Test project pgumeson/ff-track-ref-pipeline-creation created: one CI job plus manual padding jobs that never run, so pipeline creation takes long enough for auto-merge to race it. "Pipelines must succeed" off
  • Baseline recorded with the flag off: push options with auto-merge (pgumeson/ff-track-ref-pipeline-creation!10) and the merge API with auto_merge=true on a prepared MR (pgumeson/ff-track-ref-pipeline-creation!11) both merged before the pipeline was created
  • Kibana baselines recorded (7 days)
  • Heads-up posted in #g_pipeline-execution_engineering and #support_gitlab-com

Non-production

/chatops gitlab run feature set track_ref_pipeline_creation true --dev --pre --staging --staging-ref
  • Enabled on non-production (dev, staging, staging-ref, pre); no soak, the production project gate is the first real check

Production, project gates

/chatops gitlab run feature set --project=pgumeson/ff-track-ref-pipeline-creation track_ref_pipeline_creation true
/chatops gitlab run feature set --project=gitlab-org/gitlab track_ref_pipeline_creation true
  • Test project: both scenarios now wait for the pipeline and merge after it passes
  • gitlab-org/gitlab enabled for 1 hour, Kibana checks flat

Production, percentage of actors (projects)

/chatops gitlab run feature set track_ref_pipeline_creation 10 --actors
/chatops gitlab run feature set track_ref_pipeline_creation 50 --actors
/chatops gitlab run feature set track_ref_pipeline_creation true
  • 10% for 1 hour, Kibana checks flat
  • 50% for 1 hour, Kibana checks flat
  • 100%

Before global rollout

  • Docs and version history: not needed, gitlab_com_derisk flags are not documented
  • Breaking changes: none
  • Change management issue: not required, the change is reversible with one command and touches no stored data
  • External API consumers: none depend on the flag

Verification

  • Checks flat through the first full business day at 100% and again before the removal MR merges
  • Race signal: merged MRs with auto-merge whose head pipeline was created after merged_at, the days at 100% against the same weekdays before, split by branch and merge request pipelines
  • Verification items ticked on gitlab-com/gl-infra/production-engineering#29150 (closed)

Cleanup

Remove the flag and its YAML definition. That MR carries Changelog: fixed, since it is what releases the fix to self-managed and Dedicated. It lands in 19.7: every node needs the Sidekiq argument shape and event schema from !256819 (merged) (shipped in 19.6) before the behaviour is always on, so the flag stays off by default through 19.6 and the MR merges only after the 19.6 stable branch is cut on 2026-11-17. The flag stays enabled on GitLab.com until the removal is deployed. Then:

/chatops gitlab run release check <merge-request-url> 19.7
/chatops gitlab run feature delete track_ref_pipeline_creation --dev --pre --staging --staging-ref --production
  • Removal MR !268505 merged, after 19-6-stable-ee exists
  • Release check: not in 19.6, in 19.7; flag deleted
  • Close this issue, then #29150 and #595420

Rollback

/chatops gitlab run feature set track_ref_pipeline_creation false                                         # production
/chatops gitlab run feature set track_ref_pipeline_creation false --dev --pre --staging --staging-ref     # non-production
Edited by Peter Gumeson