Duplicate child pipelines being created

This bug has been caught on staging

When trying to create a child pipeline we seem to raise an error and the Ci::CreateCrossProjectPipelineWorker retries the operation, creating duplicate pipelines.

Errors found on Kibana

Error: Ci::Pipeline::BridgeStatusError

Investigation

The error is raised if the bridge is not active when calling update_bridge_status!. The bridge may not be active because it could be dropped if errors occur when creating a downstream pipeline - introduced in !24092 (merged) and modified also in !24735 (merged) - /cc @matteeyah

In !24665 (diffs) we increased the number of times that update_bridge_status! can be called. - @dosuken123

I have the feeling that the 2 changes together are causing this issue.

Tests

I've ran different tests and observed the followings:

• using strategy:depend we see the bridge job failing immediately while the child pipeline works correctly
• without strategy:depend we see the bridge job passing immediately but creates duplicate child pipelines. All these child pipelines have the same bridge as source.

The primary challenge is that I could not reproduce this on my local environment, neither using the latest master nor the current staging version.

I did reproduce it consistently on staging.

Solutions

• As a quick fix we could avoid raising Ci::Pipeline::BridgeStatusError and instead ignore the operation. This would not retrigger the workers, making things worse.
• We then need to understand exactly under what circumstances this can fail.
• We should enforce the constraint that a bridge job can source only 1 pipeline.