AI Architecture v2

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

There are several ongoing efforts to define the next iteration of AI Architecture at GitLab:

• Custom Models: Susie is asking about whether a self-managed version of the AI Gateway is needed to do a PoC for Custom Models.
  • Issue for Custom Models Architecture is here.
• Update to AI Gateway Blueprint: David is iterating on the existing AI Gateway blueprint, which is out of date and no longer reflects our future plans related to Custom Models and AI Agents here.
  • Side note: the architecture diagram is also being reformatted via this MR but in the docs rather than the blueprint.
  • Jessie will be DRI for making sure that the existing blueprint is updated to reflect the current status of the AI Gateway.
• Another vision for v2 of AI Gateway Blueprint: Shekhar put together a vision of AI Gateway architecture as it relates to AI Agents and Custom Models here.
  • He recorded a video of a walkthrough of the architecture here.
  • Overall the architecture there makes sense, but assumes that all AI requests come from an IDE. For Chat, this is not always the case. As as result, I want to discuss what he thinks the flow of requests would look like for an AI Feature that is part of the monolith. The flow of having the AI Gateway reach out to the monolith for AI Agent data and tool execution makes sense when the request originates from an IDE. But if the request is in the monolith (IE: chat), I would assume we’d connect from the monolith to the AI Gateway and not require any round trips back to the monolith. But this flow is not represented in Shekhar’s flow diagram right now.
• Product's role in AI Architecture plans: Eduardo and I discussed the need for not just an architecture blueprint, but a higher-level vision for how all of the ongoing AI work fits together. This is more of a product exercise than an engineering exercise, but is important for engineering to know if the architecture blueprints we are putting together are moving us toward the larger product vision. This will become the main “AI” blueprint for all things AI at GitLab, and each subsequent technical decision can be a decision record related to that Blueprint. For example, the existing AI Gateway blueprint will become a decision record.
  • Eduardo will be DRI for completing this broader technical vision blueprint.
  • Having this larger vision will help us prioritize related work, such as adopting AI Agents for Duo features.
  • Once the vision doc is merged, we can merge Shekhar’s blueprint into the existing AI Gateway blueprint/decision record. There may be some changes to what he has today because on the outcome of the vision doc.