<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=545857) </details> <!--IssueSummary end--> In the last week we've had 2 incidents that broke `gprd-cny` due to changes that contained both frontend and backend updates touching GraphQL getting merged. * https://app.incident.io/gitlab/incidents/1265 caused by https://gitlab.com/gitlab-org/gitlab/-/merge_requests/191475 * https://app.incident.io/gitlab/incidents/1276 caused by https://gitlab.com/gitlab-org/gitlab/-/merge_requests/185081 These types of updates break because of how the canary environment works - API requests are routed to canary backends based on [these paths](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/blob/master/roles/gprd-base-haproxy-main-config.json#L109-124). However, canary frontend requests using GraphQL cannot be routed this way since they all go to the same path (`/api/graphql`). In our rolling deployments, canary frontend will receive a new version of the code with GraphQL schema changes, but the main backends won't receive it until the main stage deployment is completed. Since 5% of requests are randomly routed to canary, there's actually a 95% chance of a frontend request from canary failing due to incompatible GraphQL schemas between the frontend and backend. We currently have a dangerbot warning that detects such changes, e.g. https://gitlab.com/project_278964_bot_b66b169fda2a3223a645094be35d5515 {width=602 height=182} However this is clearly not effective enough at preventing these types of breakages. It doesn't help that the message is buried under a bunch of other warnings. This issue is to investigate what other measures we can take to prevent these changes from getting merged. _This ticket was created from_ [_INC-1276_](https://app.incident.io/gitlab/incidents/1276) _and was automatically exported by_ [_incident.io_](https://app.incident.io) :fire: