Qualification testing for AI Gateway on EKS

Objective

Confidence that AI Gateway running on AWS EKS is fit to handle production traffic.

Background

We have AI Gateway up and running in two EKS clusters, one for staging and one for production. The instances are accessible via:

• Production: https://app.ai-gateway-eks.us-east-1.aws.runway.gitlab.net/
• Staging: https://app.ai-gateway-eks.us-east-1.aws.staging.runway.gitlab.net/

The instances are configured to connect to GLGO running on EKS via the AIGW_GLGO_BASE_URL environment variable:

• Production: https://auth.aws.gitlab.com
• Staging: https://auth.aws.staging.gitlab.com

Test cases

• ensure AI Gateway accepts valid authentication headers
• ensure communication with Amazon Q works
• ensure other AI backends work
  • Vertex AI
  • Anthropic
  • Fireworks
• more …

Set up

Unfortunately the AI Gateway documentation only outlines how to test locally (with self-signed certificats), which is different from what we need to do here. We'll need input from the AI Gateway team on how to do this.

Follow up

• A trailing slash in the AI Gateway URL setting ... (gitlab-org/gitlab#573787)