Rollout plan to switch from our model to Google for customers with 3rd party AI enabled
Current state
Code suggestions are routed to Google for one test project. So far the user feedback and performance has been quite favorable: Test GitLab code suggestions on the Google models (#148 - closed)
Phase 1
Completed on Jun 14
-
Enable code suggestions for GitLab projects (beyond the one test project currently enabled) - @achueshev @stanhu -
Announce internally and monitor for feedback Test GitLab code suggestions on the Google models (#148 - closed) @wayne -
Monitor performance, error rates, and code suggestion user acceptance rate - @marin and team for infrastructure, @mray2020 and team for AI Assist
Phase 2
ETA: Starts Jun 20, completes Jun 22
-
Doc changes are deployed Initial docs for third-party AI for code sugges... (gitlab-org/gitlab!123334 - merged) DRI: @nkhalwadekar -
Add third party feature link to toggle (gitlab-org/gitlab!123685 - merged) merged and deployed DRI: @m_gill -
Complete work to allow for A/B testing so we can do a phased rollout of using the Google model: A/B/C Testing for Code Suggestion (#150 - closed) DRI @mray2020 - ETA Jun 19 -
Turn on the feature flag for the in product messaging, informing customers that they will be switching to 3rd party AI if they have 3rd party AI enabled https://gitlab.com/gitlab-org/gitlab/-/issues/415188+ - DRI @kniechajewicz -
Jun 20: Phase rollout to send code suggestions to Google for 25% of requests DRI @mray2020 -
Monitor performance, error rates, and code suggestion user acceptance rate at 25% - @marin and team for infrastructure, @mray2020 and team for AI Assist -
Jun 21: Phase rollout to send code suggestions to Google for 100% of requests DRI @wayne -
Monitor performance, error rates, and code suggestion user acceptance rate at 100% - @marin and team for infrastructure, @mray2020 and team for AI Assist
Edited by Wayne Haber