Move CancelPendingPipelines step to run async

Problem

The step CancelPendingPipeline of the pipeline creation ranges from 6s to 19s. This time is spent looking for previous pipelines that are pending or running on the same ref where the new pipeline has just been created. Then cancel these pipelines one by one, including any child pipelines.

image

This is something we should be able to do asynchronously.

Solution

Move the logic to a Sidekiq worker and ensure that the CancelPendingPipeline step duration is reduced to nearly 0s.

Expected benefit

15-20% reduction in pipeline creation time.

Edited by Fabio Pitino