Keep track of the failures on stable branches
Context
Delivery is working to extend the GitLab maintenance policy to support bug fixes to the two previous monthly releases in addition to the current stable release. Part of this work will open up the stable branches to allow developers to merge bug fixes directly into the stable branch rather than rely on the pick labels.
Stable branches represent the source of GitLab releases, because of this, it's vital to guarantee their readiness and availability. There are efforts planned to minimize the failures on these branches (such as #2725 (closed) and #2654 (closed)), however, when failures occur they should be promptly addressed to prevent blocking any release schedule.
Two types of failures have been identified so far:
- Legit failures introduced by merge requests.
- Flaky failures present on stable branches.
This issue focuses on the former, the latter is being worked on #2637 (closed)
Proposal: Keep track of the legit failures on stable branches
- Failures on stable branches should be communicated:
- A Slack message on the
#releases
channel should be automatically posted (This is being worked on #2775 (closed))
- A Slack message on the
- An issue on the
release/tasks
repository should be opened notifying these failures. - The issue should be assigned to the merge request author and maintainer and ping the release managers.
- The issue should:
- Describe the branch affected, the failure, and the merge request that introduced the failure
- Guidance should be offered to mitigate the issue:
- Search for a similar broken-master issue - If one exists, then the process should be to ask the DRI of the broken master issue to cherry-pick it into stable branches
- If the MR author or maintainer is not available, suggest using dev-on-call process for this.
- The issue should use have the release-blocker label assigned.
- Care should be taken of not opening the same issue multiple times