Monitor tag and publish pipelines during a patch release

Summary

When we create the tags for a patch release, pipelines are automatically started for those tags. release-tools does not currently monitor those pipelines for failures, nor do we have a step in the patch release template for RMs to monitor those pipelines.

The result is that we are not aware when pipelines fail, causing some versions to never be published. This has happened a few times now. Usually we only notice once a customer asks about a missing package. Recent example: https://gitlab.com/gitlab-org/release/tasks/-/issues/11350#note_1969701578.

Proposal

Automating the monitoring of the pipelines would be ideal, but is not trivial since the pipelines run in dev.gitlab.org, while release-tools runs in gitlab.com or ops.gitlab.net.

Though it's not ideal, the boring solution is to add a step to the patch release template for RMs to monitor the tag pipelines. This is better than what we have now.

• List out the tag pipelines in Slack or in the logs of the chatops CI job for tagging new versions. Currently, we print a list of pipelines in the CI job logs of the publishing chatops command. We can also do it when tagging new versions.
• Add a step to the patch release template, after running the tagging commands, for RMs to make sure the tag pipelines succeed. The pipelines usually go into the maual/blocked state since there are manual publishing jobs at the end of the pipeline.
• When publishing packages, do not ignore pipelines that have failed. release-tools only automatically plays manual jobs (publish_service/base_publish_service.rb#L45). If a previous job in the pipeline has failed, the manual publishing job will be ignored since it will not be eligible for playing yet. Instead, any pipelines that are failed should be listed for RMs to check. - gitlab-org/release-tools!3241 (merged)
• Add a step to the patch release template for RMs to make sure that the publishing jobs succeed. - gitlab-org/release-tools!3264 (merged)