Since the start of version 15, we're noticing issues with tags not building pipelines

Problem Statement

During the publish stage, release-tools will push the tag for a package for omnibus and CNG to the appropriate repos on the dev instance to begin the build process. Two releases in a row, for versions 15.x we've suffered some sort of failure that prevents those tags from building.

Evidence for this can be seen as part of incident: gitlab-com/gl-infra/production#7303 (closed)

And we ran into this again on omnibus for versions 15.1.1 and 15.0.4 (internal slack thread): https://gitlab.slack.com/archives/C1FCTU4BE/p1656595801108669

And another thread with some additional context: https://gitlab.slack.com/archives/CCFV016SV/p1656593583941719

Release tools is not throwing an error, and the tags exist, but the pipelines for those tags are not being scheduled it appears.

This is causing some toil among release managers. Let's see if we can figure out what is going wrong.

Current solution

Kick off a pipeline manually using that tag as the source.

Future solution

• One current suspect is that the runner is misbehaving (sourced from above slack thread)
• ...