Duplicate pipelines created when creating MR via REST API

Summary

When creating a merge request via the REST API, it creates multiple pipelines for the same (merged result) commit

Steps to reproduce

Create a branch via API
Cherry-pick a load of commits via API

number of commits doesn't seem to correspond to number of pipelines created

Make an MR via API

payload = {
    "source_branch": new_branch,
    "target_branch": release_branch,
    "title": merge_req_title,
    "description": merge_req_body,
    "labels": ",".join(labels),
    "remove_source_branch": True,
    "squash": False,
}
response = requests.post(GITLAB_API_MERGE_REQ, headers=get_auth_header(), data=payload)

Look at MR, notice multiple pipelines

Example Project

What is the current bug behavior?

Multiple duplicate pipelines are created for the same commit - 4 or 5 thus far.

What is the expected correct behavior?

1 pipeline.

Relevant logs and/or screenshots

I went digging into application_json.log and found the following:

{"severity":"INFO","time":"2024-01-17T08:25:55.675Z","correlation_id":"XX","meta.caller_id":"NewMergeRequestWorker","meta.remote_ip":"10.12.25.34","meta.feature_category":"code_review_workflow","meta.user":"cpigott","meta.user_id":3,"meta.project":"dev/rvs/rvs","meta.root_namespace":"dev","meta.client_id":"user/3","meta.root_caller_id":"POST /api/:version/projects/:id/merge_requests","message":"Pipeline authorized","project_id":298,"user_id":3}
{"severity":"INFO","time":"2024-01-17T08:26:00.085Z","correlation_id":"XX","meta.caller_id":"MergeRequests::CreatePipelineWorker","meta.remote_ip":"127.0.0.1","meta.feature_category":"continuous_integration","meta.user":"cpigott","meta.user_id":3,"meta.project":"dev/rvs/rvs","meta.root_namespace":"dev","meta.client_id":"user/3","meta.root_caller_id":"POST /api/:version/internal/post_receive","message":"Pipeline authorized","project_id":298,"user_id":3}
{"severity":"INFO","time":"2024-01-17T08:26:02.379Z","correlation_id":"XX","meta.caller_id":"MergeRequests::CreatePipelineWorker","meta.remote_ip":"127.0.0.1","meta.feature_category":"continuous_integration","meta.user":"cpigott","meta.user_id":3,"meta.project":"dev/rvs/rvs","meta.root_namespace":"dev","meta.client_id":"user/3","meta.root_caller_id":"POST /api/:version/internal/post_receive","message":"Pipeline authorized","project_id":298,"user_id":3}
{"severity":"INFO","time":"2024-01-17T08:26:03.643Z","correlation_id":"XX","meta.caller_id":"MergeRequests::CreatePipelineWorker","meta.remote_ip":"127.0.0.1","meta.feature_category":"continuous_integration","meta.user":"cpigott","meta.user_id":3,"meta.project":"dev/rvs/rvs","meta.root_namespace":"dev","meta.client_id":"user/3","meta.root_caller_id":"POST /api/:version/internal/post_receive","message":"Pipeline authorized","project_id":298,"user_id":3}
{"severity":"INFO","time":"2024-01-17T08:26:05.419Z","correlation_id":"XX","meta.caller_id":"MergeRequests::CreatePipelineWorker","meta.remote_ip":"127.0.0.1","meta.feature_category":"continuous_integration","meta.user":"cpigott","meta.user_id":3,"meta.project":"dev/rvs/rvs","meta.root_namespace":"dev","meta.client_id":"user/3","meta.root_caller_id":"POST /api/:version/internal/post_receive","message":"Pipeline authorized","project_id":298,"user_id":3}
{"severity":"INFO","time":"2024-01-17T08:26:07.632Z","correlation_id":"XX","meta.caller_id":"NewMergeRequestWorker","meta.remote_ip":"10.12.25.34","meta.feature_category":"code_review_workflow","meta.user":"cpigott","meta.user_id":3,"meta.project":"dev/rvs/rvs","meta.root_namespace":"dev","meta.client_id":"user/3","meta.root_caller_id":"POST /api/:version/projects/:id/merge_requests","merge_request_id":1974,"message":"Pipeline created"}

which is 5 "Pipeline authorized" messages, but only a single "Pipeline created" message?

There's quite a lot of "Cannot obtain an exclusive lease" messages too:

{"severity":"INFO","time":"2024-01-17T08:26:17.608Z","correlation_id":"XX","meta.caller_id":"PipelineProcessWorker","meta.remote_ip":"10.12.25.34","meta.feature_category":"continuous_integration","meta.user":"cpigott","meta.user_id":3,"meta.project":"dev/rvs/rvs","meta.root_namespace":"dev","meta.client_id":"user/3","meta.root_caller_id":"POST /api/:version/projects/:id/merge_requests","message":"Cannot obtain an exclusive lease. There must be another instance already in execution.","lease_key":"ci/pipeline_processing/atomic_processing_service::pipeline_id:15460","class_name":"Ci::PipelineProcessing::AtomicProcessingService","lease_timeout":60}
{"severity":"INFO","time":"2024-01-17T08:26:21.981Z","correlation_id":"XX","meta.caller_id":"PipelineProcessWorker","meta.remote_ip":"10.12.25.34","meta.feature_category":"continuous_integration","meta.user":"cpigott","meta.user_id":3,"meta.project":"dev/rvs/rvs","meta.root_namespace":"dev","meta.client_id":"user/3","meta.root_caller_id":"POST /api/:version/projects/:id/merge_requests","message":"Cannot obtain an exclusive lease. There must be another instance already in execution.","lease_key":"ci/pipeline_processing/atomic_processing_service::pipeline_id:15460","class_name":"Ci::PipelineProcessing::AtomicProcessingService","lease_timeout":60}
{"severity":"INFO","time":"2024-01-17T08:26:24.166Z","correlation_id":"XX","meta.caller_id":"PipelineProcessWorker","meta.remote_ip":"10.12.25.34","meta.feature_category":"continuous_integration","meta.user":"cpigott","meta.user_id":3,"meta.project":"dev/rvs/rvs","meta.root_namespace":"dev","meta.client_id":"user/3","meta.root_caller_id":"POST /api/:version/projects/:id/merge_requests","message":"Cannot obtain an exclusive lease. There must be another instance already in execution.","lease_key":"ci/pipeline_processing/atomic_processing_service::pipeline_id:15460","class_name":"Ci::PipelineProcessing::AtomicProcessingService","lease_timeout":60}

(I've got the full log from that time period if you deem it necessary)

My guess is some sort of race condition when creating the MR, but I've no idea really. Needless to say, ordinarily when you create a MR through the UI only a single (merged result) pipeline is created.

Output of checks

N/A

Results of GitLab environment info

16.7

N/A

Results of GitLab application Check

N/A