## Problem Space Currently all GitLab Duo Features are served by hosted models accessed via an API that is routed through the AI Framework. There is currently no support of locally hosted models (local inference) in support of GitLab AI features. This precludes certain customers from accessing and leveraging AI Duo Features. ## Proposals We seek to make models available in offline environments or run locally. This may refer to a local models running physically on an engineer's machines, but has more value when extended to connect to models running anywhere within an enterprise, including airgapped configurations. ### Proposal/Exploration #1 - Determining OS Models to Support Local Inference 1. Identify and begin validation of OS models that could potentially serve DuoFeatures to include: code generation / suggestion, Chat, and Explain this Vulnerability. 2. Analyze the architecture and parameters of the potential model, considering factors that may affect local compute requirements, including the number of layers, the size of the model parameters, and the computational complexity of individual operations (matrix multiplications, convolutions), and inference requirements. 3. Carry out benchmarking experiments to measure the model's inference speed and resource usage on your target hardware/compute resources, or a variety of compute resource. 4. Profile/analyze the resource usage of the model during inference (ie [TensorFlow ](https://www.tensorflow.org/guide/profiler)tool suite) 5. Assess scalability 6. Determine minimum hardware requirements ### Proposal/Exploration #2 - Explore/Determine Infrastructure for Configuring Duo with Locally-Hosted OS Models 1. research/explore options for packaging the model configurations, dependencies, and configuration * [omnibus](https://docs.gitlab.com/omnibus/architecture/)? * explore Onyx formats for local model deployment as demonstrated by [this POC](https://www.youtube.com/watch?v=kDEz8ZuKX58) 2. configuring the model/internal API- requirements for different GL models (self-service, airgapped, dedicated) 3. Gitlab CI/CD configuration requirements 4. access controls 5. AI Gateway requirement? ### Proposal/Exploration #3 - Enterprise-Hosted AI Gateway Traversing via a local AI Gateway allows the enterprise to store authentication secrets (etc) in the AI Gateway, without having to distribute them to all developers, as well as perform logging and observability. > Consuming application (eg: IDE) -\> AI Gateway -\> Enterprise-hosted Model 1. collaborate with AI Framework to determine requirements for creation/deployment of local AI Gateway 2. related work 1. [AI Gateway as the Sole Access Point for Monolith](https://gitlab.com/groups/gitlab-org/-/epics/13024) ## Target Hardware - Apple Silicon - M1-M3 - Nvidia - GeForce RTX 30-40 series - AMD - Radeon RX 6000 - 7000 series