Breaking changes to runbooks causing production alerts and maintenance window delays

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Label this issue

Summary

An unplanned breaking contract change was made in the Runbooks repo, this made it to production on GitLab Dedicated causing a number of deployments to break.

Production Incidents

• https://app.incident.io/gitlab/incidents/12701
• https://app.incident.io/gitlab/incidents/12862
• https://app.incident.io/gitlab/incidents/12867
• https://app.incident.io/gitlab/incidents/13099

Impact

• Slower than usual maintenance windows, with a high chance of affecting customers.
• Heavy after hours toil for GitLab Dedicated EOCs.
• Mismatched documentation, causing a lot of confusion, affecting our ability to remediate the problems quickly.

Setting severity 3 due to high chance of repeated customer impact, and high toil.

Recommendation

Implement checks for breaking changes so that it becomes explicit. A breaking change needs to be a new major version, with updated docs, release notes, and a strategy to upgrade prod in advance.

Verification

Review by the Dedicated team.