AI Gateway Helm chart OIDC provider selection when AIGW_CUSTOMER_PORTAL_URL not set differs from docker deployment and expected selection

Summary

When deploying a self-hosted model a local AI Gateway must be deployed. This can be deployed in docker or in k8s using the Helm chart.

AI Gateway uses OIDC Discovery for authenticating incoming requests, and by default tries to use the AI_GATEWAY_URL and https://customers.gitlab.com as OIDC providers.

You can override https://customers.gitlab.com with a custom URL by setting the AIGW_CUSTOMER_PORTAL_URL environment variable. The AI_GATEWAY_URL is always used.

When deployed using docker, and the AIGW_CUSTOMER_PORTAL_URL environment variable is not set in the container config, the two default providers mentioned above are used. Inside the AIGW container, as expected, the AIGW_CUSTOMER_PORTAL_URL variable is not defined.

However, when deployed using the Helm chart, and the AIGW_CUSTOMER_PORTAL_URL environment variable is not set in the chart config, a variable still gets defined in the container with an empty value. This causes https://customers.gitlab.com to be overridden with '' and a no scheme supplied error is logged during OIDC discovery. The AI_GATEWAY_URL is still used.

This means the default provider selection differs between the docker and Helm AIGW deployments, and may lead to unexpected behaviour and confusion.

Reproduced with AIGW Helm chart version 0.3.0, app version self-hosted-v17.9.0-ee.

Workarounds

You can workaround this by setting customerPortalUrl: https://customers.gitlab.com in the chart config.

Another workaround is to modify the ai-gateway configMap created when the chart is deployed, and remove the AIGW_CUSTOMER_PORTAL_URL: "" entry. When the AIGW pod is deleted and a new pod starts, it does not have the AIGW_CUSTOMER_PORTAL_URL variable defined and the default OIDC discovery occurs the same as with the docker deployment.

Steps to reproduce

  1. Install the AIGW Helm chart without setting the customerPortalUrl option.
  2. Tail the AIGW logs while running the Duo healthcheck - observe the following error:
{
  "status_code": null,
  "exception_class": "MissingSchema",
  "backtrace": "Traceback (most recent call last):
    File \"/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.11/lib/python3.11/site-packages/gitlab_cloud_connector/providers.py\", line 309, in _fetch_well_known
      res = requests.get(url=url, timeout=REQUEST_TIMEOUT_SECONDS)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File \"/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.11/lib/python3.11/site-packages/requests/api.py\", line 73, in get
      return request(\"get\", url, params=params, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File \"/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.11/lib/python3.11/site-packages/requests/api.py\", line 59, in request
      return session.request(method=method, url=url, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File \"/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.11/lib/python3.11/site-packages/requests/sessions.py\", line 575, in request
      prep = self.prepare_request(req)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
    File \"/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.11/lib/python3.11/site-packages/requests/sessions.py\", line 484, in prepare_request
      p.prepare(
    File \"/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.11/lib/python3.11/site-packages/requests/models.py\", line 367, in prepare
      self.prepare_url(url, params)
    File \"/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.11/lib/python3.11/site-packages/requests/models.py\", line 438, in prepare_url
      raise MissingSchema(
  requests.exceptions.MissingSchema: Invalid URL '/.well-known/openid-configuration': No scheme supplied. Perhaps you meant https:///.well-known/openid-configuration?
  fetch_well_known failed  ",
  "extra": {},
  "correlation_id": "01JMNTYVDXZR2FC5ST20GAVGEW",
  "logger": "cloud_connector",
  "level": "error",
  "type": "mlops",
  "stage": "main",
  "timestamp": "2025-02-22T03:24:32.960558Z",
  "message": "Invalid URL '/.well-known/openid-configuration': No scheme supplied. Perhaps you meant https:///.well-known/openid-configuration?"
}

Example Project

What is the current bug behavior?

The ai-gateway configmap created by the Helm chart sets AIGW_CUSTOMER_PORTAL_URL: "" which is treated as an explicitly set value.

What is the expected correct behavior?

An empty AIGW_CUSTOMER_PORTAL_URL value should be treated the same as the variable being undefined.

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
(For installations with omnibus-gitlab package run and paste the output of: \\\`sudo gitlab-rake gitlab:env:info\\\`) (For installations from source run and paste the output of: \\\`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production\\\`)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of: \`sudo gitlab-rake gitlab:check SANITIZE=true\`) (For installations from source run and paste the output of: \`sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true\`) (we will only investigate if the tests are passing)

Possible fixes

Edited by Justin Farmiloe