Manual intervention required when MR status is ci_must_pass

Siddharth Kannanrequested to merge
sk/update-manage-components-raise-error into master

What does this MR do and why?

Fix gitlab-com/gl-infra/production-engineering#26680 (closed)

The Gitaly update MR gitlab!188912 (closed) had a merge request status of ci_must_pass. This is something that we do not explicitly handle at the moment within the Updater class: we don't fail the CI job and a notification is sent only to the Gitaly team.

So, it fell through the cracks and no notification was sent to #f_upcoming_release. RMs were not aware that Gitaly version was not being updated and this continued for about 32 hours. This delayed the time that it took to respond to the incident gitlab-com/gl-infra/production#19691 (closed).

The notification was sent to the Gitaly team after 32 hours because we only send a notification when an MR is stale (i.e. older than 24 hours). However, once the MR has reached the ci_must_pass status, manual intervention is always required to get the MR merged. So, the managed component owner team can be informed earlier about it. This should speed up the response process and reduce the time that it takes new versions of Gitaly to reach production.

This is also related to making the updater class smarter (tracked in gitlab-com/gl-infra/delivery#20867).

More information about my investigation is here: gitlab-com/gl-infra/production-engineering#26680 (comment 2466050059).

Author Check-list

  • Has documentation been updated?
Edited by Siddharth Kannan

Merge request reports