Bridge job fails to create a new pipeline with error "Reference not found"

Summary

See also internal ticket link: https://gitlab.zendesk.com/agent/tickets/254559

When using Pipelines for Merged Results, a customer's facing intermittent failures when one of their jobs attempts a trigger: operation (to generate a child pipeline).

The bridge job that's expected to generate the child pipeline fails with the error "downstream pipeline can not be created, Reference not found"

Retrying the pipeline a few times appears to work some of the times so the issue has been intermittent

This is severe for them because its not possible to retry bridge jobs, and they're having to add dummy commits to the MR to generate new pipelines that retry

A suspicion under investigation is that this may be occurring in the following way given that their project has Pull Mirroring configured from a Bitbucket Server source:

1. MR is opened, it generates the refs/merge-requests/:iid/merge ref
2. Pipeline begins to run its jobs against refs/merge-requests/:iid/merge

• Note that internally, for regular jobs under a pipeline, the persistent refs/pipelines/:id gets used instead

1. Pipeline has a lot of jobs and it's overall runtime is fairly wide
2. Before the Pipeline has decided to execute the Bridge job, a pull mirroring update runs alongside (due to updates to /projects/:id/mirror/pull API or because 30 minute mark has passed)
3. This mirror update operation ends up clearing out all refs/merge-requests/*/* refs
4. The pipeline finally reaches execution of the bridge job
5. Bridge job runs, tries to launch a new pipeline against the same ref as the parent pipeline
6. Bridge job discovers that the MR's refs/merge-requests/:iid/merge ref is no longer present
7. Bridge job aborts with the error Reference not found

We're trying to confirm this by attempting to pause the mirroring and measuring the retention of the refs/merge-requests/:iid/merge refs via command git ls-remote origin 'refs/merge-requests/*/merge' for MRs that remain open

Steps to reproduce

I've not been able to reproduce it yet, but I have a private self-managed instance where I've attempted to setup a similar pipeline as the customer.

Please reach out on Slack for access to the self-managed instance (Slack ID: @hchouraria)

The trigger job that fails is a normally defined one:

test:
  stage: test
  rules: …
  trigger:
    include:
      - project: group/central-pipeline-configs-project
        file: path/to/a/file.yml
        ref: xyz
    strategy: depend

Example Project

What is the current bug behavior?

• Create a feature branch and add some commits to it
• Open a merge request against this branch into main
• Merge request kicks off a Pipeline for Merged Results
• A bridge job starts in this Pipeline for Merged Results due to presence of a trigger: type job
• The bridge job fails to create pipeline "downstream pipeline can not be created"
• The bridge job's error message indicates "Reference not found"

What is the expected correct behavior?

• Bridge job always creates the pipeline

Relevant logs and/or screenshots

Please see ticket https://gitlab.zendesk.com/agent/tickets/254559 for this

Output of checks

This bug happens on GitLab.com

Possible fixes

The error appears to originate here: https://gitlab.com/gitlab-org/gitlab/blob/e65b407a43911e64802fbdd2851edd69b03282f5/lib/gitlab/ci/pipeline/chain/validate/repository.rb#L11-14

The ref in the context of the Pipeline for Merged Results would be something such as refs/merge-requests/:mr_iid/merge which should always exist as long as the MR is open