Create pipeline creation requests at the start of PostReceive (FF)

What does this MR do and why?

This is a re-roll of !231843 (closed)

On push, pipeline creation requests serve as a synchronization mechanism to prevent premature merges. There were two problems with the previous approach:

1. Only MR pipelines got request records in redis. This left the race unaddressed for branch pipelines.
2. It was possible to set auto-merge before the requests were created, leading to an immediate merge instead of waiting for the pipeline to succeed. Push options were especially prone to this.

This change aims to address both issues:

1. We introduce a new ref-type pipeline creation request, and incorporate that into the mergeability check.
2. We move pipeline creation request registration to the earliest points possible of post-receive, including special handling for push options.

The fix is behind the flag track_ref_pipeline_creation, disabled by default.

References

See #595420

Screenshots or screen recordings

Scenario	Before	After
Branch pipelines
MR pipelines
Both

How to set up and validate locally

1. Add an artificial delay to post receive pipeline creation to help trigger the race being fixed. Example (with a 30 second delay):

   diff --git a/app/services/ci/create_pipeline_service.rb b/app/services/ci/create_pipeline_service.rb
   index 6cc6a48965ea..4f178f6aac43 100644
   --- a/app/services/ci/create_pipeline_service.rb
   +++ b/app/services/ci/create_pipeline_service.rb
   @@ -146,6 +146,8 @@ def execute(
        # rubocop: enable Metrics/ParameterLists, Metrics/AbcSize
    
        def execute_async(source, options)
   +      sleep 30
   +
          # Use pre-created request if available, otherwise create a new project request
          pipeline_creation_request = params[:pipeline_creation_request] ||
            ::Ci::PipelineCreation::Requests.start_for_project(project)
   diff --git a/app/services/merge_requests/create_pipeline_service.rb b/app/services/merge_requests/create_pipeline_service.rb
   index b58eea68fa96..f0584d5e6dcc 100644
   --- a/app/services/merge_requests/create_pipeline_service.rb
   +++ b/app/services/merge_requests/create_pipeline_service.rb
   @@ -12,6 +12,8 @@ def execute(merge_request)
        end
    
        def execute_async(merge_request)
   +      sleep 30
   +
          pipeline_creation_request = params[:pipeline_creation_request] ||
            ::Ci::PipelineCreation::Requests.start_for_merge_request(merge_request)

2. Save this script to a file and add it to your PATH. The rest of the instructions will refer to this as the command create-mr-with-auto-merge-enabled

   #!/usr/bin/env zsh
   set -eou pipefail
   
   file=file-$RANDOM
   branch=branch-$RANDOM
   
   git checkout main 
   git pull
   git checkout -b "${branch}"
   touch "${file}"
   git add "${file}"
   git commit -am "Add ${file}"
   git push -u origin "${branch}" \
       -o merge_request.create \
       -o merge_request.target=main \
       -o merge_request.auto_merge

3. Check out a project from your GDK locally. We will use two rudimentary .gitlab-ci.ymls to test the two most common workflows:

   • Branch pipelines:

     # .gitlab-ci.yml
     job:
       script: exit 0

   • Merge request pipelines:

     # .gitlab-ci.yml
     workflow:
       - rules: '$CI_MERGE_REQUEST_IID || $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
     
     job:
       script: exit 0

4. Run create-mr-with-auto-merge-enabled, visit the merge request link from the push response message

5. Observe what happens to the merge request

Expected results:

• track_ref_pipeline_creation disabled: The created MR gets merged before the pipeline is even created
• track_ref_pipeline_creation enabled: The created MR waits for the pipeline to be created

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.