Document a workflow on how feature changes with an accompanying project settings modification should be handled

Problem

Related to this incident: gitlab-com/gl-infra/production#3010 (closed). An MR was merged and released to staging, which modified features but also admin project settings. The latter ended up impacting our Staging Gitaly, by triggering a code path that ended up calling Gitaly for each MR in the MRs index page to evaluate MR approvers.

The Project Settings change and its impact were not obvious, and it took us (devs and infra together) a number of hours to understand what had happened.

Proposal

It is clear that Project Settings changes can impact how Gitlab works/performs, drastically, as seen in this incident.

Therefore we propose we make this a new case subject to our Change Management Process, similar to how we handle Feature Flag changes today: Every time we change some Project Settings in Staging or Production, meeting specific criteria (to define), it should be done accompanied by a change management issue following our Change Request Workflow.

Edited by Alberto Ramos