Trace incomplete deployment pipelines
Summary
Currently, we only trace completed pipelines. This is because the traces are generated by a job at the end of the pipeline. We cannot simply run the job on failure, since many failures are transient and succeed on retry.
Proposal
- Have a manual job at the end of the coordinator pipeline that traces the same pipeline. We can convert the existing
trace-pipeline
job to be manual. - Have another job just before the
trace-pipeline
job.- This job can be in the same stage as the
trace-pipeline
job. - This job should trigger the manual
trace-pipeline
job of the same pipeline. - This job should also trigger the
trace-pipeline
jobs of previous coordinator pipelines. If any of the jobs have already been triggered, they should be ignored (don't throw an error).- We can loop through previous pipelines that are in failed or blocked or canceled status, and stop when we encounter a successful pipeline.
- This job can be in the same stage as the
Once a coordinator pipeline completes, any previous pipelines cannot be retried/continued any way. So we can run the trace-pipeline
job of incomplete pipelines if a pipeline after that has completed.
Edited by Reuben Pereira