Support a mix of self-hosted and GitLab-hosted models for Duo features
In the current architecture, there are four possible configurations available for the AI Gateway
| Instance Type | AI Gateway | Models accessible |
|---|---|---|
| .com | .com hosted | GitLab managed |
| Dedicated | .com hosted | GitLab managed |
| Self-Managed | .com hosted | GitLab managed |
| Self-Managed | Self-Hosted | Self-hosted models |
| Self-Managed Airgapped | Self-Hosted | Self-hosted models |
GitLab managedmodels means the Cloud hosted models at Anthropic and Vertex, accessible using the GitLab API key.
For all configurations except those that are both self-hosted AND airgapped, the default GitLab model is selected for each feature. On airgapped instances, the feature is not available to the customer if there is no supporting self-hosted model. Users on non-airgapped instances (using the .com AI Gateway) cannot currently select a mix of self-hosted and GitLab managed models.
Proposal
Allow a local AI Gateway to optionally access GitLab hosted models via configuration. This will allow customers who have deployed their own local AI Gateway to selectively chose GitLab managed models at the Duo Feature level.
| Instance Type | AI Gateway | Models accessible |
|---|---|---|
| .com | .com hosted | GitLab managed |
| Dedicated | .com hosted | GitLab managed |
| Self-Managed | .com hosted | GitLab managed |
| Self-Managed | Self-Hosted | Self-hosted models + GitLab managed |
| Self-Managed Airgapped | Self-Hosted | Self-hosted models |
This modularity would be limited to specific supported use-cases, where there are compelling privacy reasons for customers to utilize hybrid approaches. To start with, we would support self-hosted embedding models.
Investigation is needed around if the local AI Gateway should point to the .com AI Gateway, or be able to access the GitLab managed models directly (would need API keys to do this).