Coordination for enabling code suggestion banner in controlled increments
This issue is to be used for coordination between different stakeholders to enable the code suggestions banner through code_suggestions_alert
feature flag.
In the AI strategy discussion for 2023-05-31, questions were raised around when we will be able to start inviting new traffic to increase Code Suggestion usage.
While the corrective actions are being worked on in parallel in Code Suggestions corrective actions MVC (#125 - closed), there was a suggestion to do a smaller percentage increase than we did before (we did 100% and then 10%) as a means to validate the work that has been done so far, and also get a better insight in what are the numbers of requests that we can support.
The AI Assist and Infrastructure teams would be in full control of the percentage they are comfortable with, and they would require no approvals for changing percentages. Disabling the banner can be done at any time if the load on the system is too high.
The only requirement is to note down when the percentage change has been set via the feature flag, in order to enable the product teams to track conversion rates precisely.
Decision
Increase the percentage of code_suggestions_alert
by 1% every six hours up until 5%, starting at 2023-06-01 at 14:00 UTC .
Use this time to monitor how the system behaves at different peak times. Ramp it down if we see that the latency is changing, and validate that we have good understanding of critical items raised in #125 (closed). We seem to have been peaking at 15 rps, so it would be good to know whether that would change in any significant way with percentage increases.
Rollout
Enable the feature flag with:
/chatops run feature set code_suggestions_alert N --actors --ignore-feature-flag-consistency-check
where N is:
- 1% set by @igorwwwwwwwwwwwwwwwwwwww at 16:09 UTC 2023-06-02
- 2% set by @stanhu at 22:05 UTC (2pm PT) 2023-06-02
- 4% set by @a_akgun at 06:51 UTC 2023-06-02
- 5% set by @marin at 13:29 UTC 2023-06-02
- 15% set by @marin at 09:18 UTC 2023-06-05
- 50% set by @marin at 17:34 UTC 2023-06-05
- 100% set by @marin at 11:49 UTC 2023-06-06
Stopping the rollout
To disable the feature flag, go to #production Slack channel and run:
/chatops run feature set code_suggestions_alert 0 --actors --ignore-feature-flag-consistency-check
Section below is in case you are wondering if you are making the right decision to disable:
Go to code suggestion dashboard. Under Aggregated Service Level Indicators (𝙎𝙇𝙄𝙨)
, look at panels that have Apdex
in their title. If they are moving from 100% towards 75% and continuing to trend down towards 50%, and the ingress SLI Error ratio
is moving from 0% towards 10% and trending up, customers are experiencing issues with the service and we should not invite more traffic. Please use the feature flag above to disable the banner.