Is it feasible to have chatops warn with a large jump in applied % for new FF?
Summary
In a recent Sev 1, there is an open question of whether the FF was enabled to 100% in one go. If it had been rolled out in steps, we might have caught the error of mismatch in repo storage size calculations when it impacted far fewer projects, greatly containing the blast radius of the incident.
We would like to know whether we can implement a warning or reminder message when new FF are enabled if it is rolled out in one or two big steps.
Related Incident(s)
Originating issue(s): gitlab-com/gl-infra/-/issues/7638
Desired Outcome/Acceptance Criteria
We will have a clear answer of whether our chatops code could support such a feature, and create an issue for that work. Timebox the investigation to no more than half a day.
Associated Services
Corrective Action Issue Checklist
-
Link the incident(s) this corrective action arose out of -
Give context for what problem this corrective action is trying to prevent from re-occurring -
Assign a severity label (this is the highest sev of related incidents, defaults to 'severity::4') -
Assign a priority (this will default to 'Reliability::P4')