Functional Testing: Configure self-hosted models on Staging Ref

Overview

Staging replicates the gitlab.com behavior that is using Cloud Connector to power AI features. Cloud Connector is a service that runs GitLab-managed AI Gateway that processes the requests from gitlab.com and self-managed customers and propagates those requests to LLMs vendored by GitLab (for example, Anthropic using Gitlab-owned API key or Vertex AI models using Gitlab-owned credentials).

Within groupcustom models we added support for configuring self-hosted models. To make it work, the customers must deploy their own AI Gateway.

Proposal

Staging Ref doesn't have AI features enabled. Configuring self-hosted models there is an opportunity to enable AI features and test them in an environment closer to production.

Challenges

• A separate AI Gateway with custom models enabled must be used
• Duo Enterprise license must be provided

Next steps

• MRs get merged:
  • feat(runway): deploy ai-gateway-custom (gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!1407 - merged)
  • https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit-configs/staging-ref/-/merge_requests/190+
  • AI Gateway will then be active
• We apply an offline license to the instance
• An admin configures self-hosted models (Staging Ref admin credentials are available to Eng Department via 1Password)
  • Any team member logs in and can go ahead with testing. However, we probably need to be informed first, so we can assign a Duo seat to the user.

Nice to have

• Perform daily evaluation runs for Duo Chat against Staging Ref. Ideally, we should receive a dashboard similar to this one.