Duplicate child pipelines being created
This bug has been caught on staging
When trying to create a child pipeline we seem to raise an error and the Ci::CreateCrossProjectPipelineWorker
retries the operation, creating duplicate pipelines.
Error: Ci::Pipeline::BridgeStatusError
Investigation
The error is raised if the bridge is not active
when calling update_bridge_status!
. The bridge may not be active
because it could be dropped if errors occur when creating a downstream pipeline - introduced in !24092 (merged) and modified also in !24735 (merged) - /cc @matteeyah
In !24665 (diffs) we increased the number of times that update_bridge_status!
can be called. - @dosuken123
I have the feeling that the 2 changes together are causing this issue.
Tests
I've ran different tests and observed the followings:
- using
strategy:depend
we see the bridge job failing immediately while the child pipeline works correctly - without
strategy:depend
we see the bridge job passing immediately but creates duplicate child pipelines. All these child pipelines have the same bridge as source.
The primary challenge is that I could not reproduce this on my local environment, neither using the latest master
nor the current staging version.
I did reproduce it consistently on staging.
Solutions
- As a quick fix we could avoid raising
Ci::Pipeline::BridgeStatusError
and instead ignore the operation. This would not retrigger the workers, making things worse. - We then need to understand exactly under what circumstances this can fail.
- We should enforce the constraint that a bridge job can source only 1 pipeline.