Consider requesting a rollout review before enabling a feature flag globally on gitlab.com

Summary

We have guidance on rolling out changed behind feature flags, including an issue template. However, a rollout is often done by a single individual, and usually the engineer who developed the feature.

We are familiar with ways that code changes can lead to incidents if the code it authored by a single individual. We have a code review process to create opportunities for someone else to contribute to the change in a way that improves the result, but also to help avoid problems.

Proposal

We could require a review of the rollout process, similar to how we review code changes. It wouldn't need to be too time-consuming. The review would involve verifying that the process was appropriate for the change, including a brief review of any pre-production testing, and a review of the duration and timing of the incremental rollout that considers the scope of the feature change and the likelihood of users encountering problems that might exist. For example, we note:

For some features a few minutes is enough, while for others you may want to wait several hours or even days.

However, that leaves a lot of room for different conclusions about what is sufficient. We could (and should) provide some more guidance, but it will still be a judgement call that could be improved by having more than one person make that judgement.

The review should be performed by someone in the same group as the change author, but who was not involved in the development. Those criteria should mean the reviewer is familiar enough with the feature that they can quickly understand what was changed, but they should also be unfamiliar with the specifics of the change and therefore able to provide a fresh perspective on anything that might have been missed.