Automate uploading of pipeline traces to GitLab Observability
Summary
We need to have successful deployment pipelines be automatically traced when they complete.
Proposal
One option is to add a job at the end of the coordinator pipeline that traces the pipeline.
Another option is to setup a webhook (in https://ops.gitlab.net/gitlab-org/release/tools) to send pipeline events to Delivery metrics. When a webhook event is received indicating that a deployment has successfully completed, delivery-metrics can trigger a job that traces the completed pipeline.
The rake task to trace a pipeline will be implemented by #2722 (closed).
Keep in mind that we might need to send traces to 3 services, including GitLab Observability. The other issues for sending traces to services are https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2613 and https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2698.
Implemented
We added a job called trace_pipeline to the end of the coordinator pipeline. The job traces the pipeline that it is running in.