Code suggestions team and architecture changes
If you are reading through issue and have architectural questions. Please refer to #161 for that discussion.
What changes are we making and why?
The existing greenfield code suggestions model and associated infrastructure has achieved much in allowing us to launch a net new service at GitLab that is based on cutting edge AI technologies.
We have decided to migrate this from a greenfield set of technologies that leveraging only our own AI model, to using standard GitLab architecture patterns and a combination of third party models and our own model in order to get the scalability, reliability, and quality we need for our customers, especially as we scale usage from current use to >100 fold higher.
We also need to test various third party models (Google, etc) for the quality of suggestions in order to choose the best models for our customers. Draft plan for this testing is documented here: https://gitlab.com/gitlab-org/gitlab/-/issues/415381+
These changes allow us to deliver on our revised roadmap.
Associated architecture changes
Migrate all needed functionality from the AI Assist model gateway to new AI Enablement abstraction layer functionality (not yet built), and change all IDEs/editors to use the abstraction layer instead. Limit changes to IDE to URL change only by building in the first iteration on abstraction layer 1:1 same endpoints. Details forthcoming on the design for this TBD from @m_gill and @timzallmann .
Move our current model / Triton from the separate GCP instance to our main GitLab product GCP instance Once the above is complete, shut down the existing production AI Assist infrastructure that is used for code suggestions. Keep any non-production portions used for model creation, testing, training and tuning. Infrastructure will transition to fully owning this environment as it is currently co-owned between infrastructure and AI Assist.
Team member borrows
- Andras and Tan will be on a borrow to AI enablement, lead by Michelle G
- Stephan and Dylan will be on a borrow to a temporary Model Testing team, lead by Sean C
- Alper will continue on the existing borrow request to the IDE extension team, lead by David O.
Alexander, Hongtao, and Bruno will continue working on ML Science, led by Mon.
Who approved this before it was decided?
Development and PM leadership.