### Summary When deploying a self-hosted model a local AI Gateway must be deployed. This can be deployed in docker or in k8s using the Helm chart. AI Gateway uses OIDC Discovery for authenticating incoming requests, and by default tries to use the `AI_GATEWAY_URL` and `https://customers.gitlab.com` as OIDC providers. You can override `https://customers.gitlab.com` with a custom URL by setting the `AIGW_CUSTOMER_PORTAL_URL` environment variable. The `AI_GATEWAY_URL` is always used. When deployed using docker, and the `AIGW_CUSTOMER_PORTAL_URL` environment variable is not set in the container config, the two default providers mentioned above are used. Inside the AIGW container, as expected, the `AIGW_CUSTOMER_PORTAL_URL` variable is not defined. However, when deployed using the Helm chart, and the `AIGW_CUSTOMER_PORTAL_URL` environment variable is not set in the chart config, a variable still gets defined in the container with an empty value. This causes `https://customers.gitlab.com` to be overridden with `''` and a `no scheme supplied` error is logged during OIDC discovery. The `AI_GATEWAY_URL` is still used. This means the default provider selection differs between the docker and Helm AIGW deployments, and may lead to unexpected behaviour and confusion. Reproduced with AIGW Helm chart version `0.3.0`, app version `self-hosted-v17.9.0-ee`. ### Workarounds You can workaround this by setting `customerPortalUrl: https://customers.gitlab.com` in the chart config. Another workaround is to modify the `ai-gateway` configMap created when the chart is deployed, and remove the `AIGW_CUSTOMER_PORTAL_URL: ""` entry. When the AIGW pod is deleted and a new pod starts, it does not have the `AIGW_CUSTOMER_PORTAL_URL` variable defined and the default OIDC discovery occurs the same as with the docker deployment. ### Steps to reproduce 1. Install the AIGW Helm chart without setting the `customerPortalUrl` option. 2. Tail the AIGW logs while running the Duo healthcheck - observe the following error: ``` { "status_code": null, "exception_class": "MissingSchema", "backtrace": "Traceback (most recent call last): File \"/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.11/lib/python3.11/site-packages/gitlab_cloud_connector/providers.py\", line 309, in _fetch_well_known res = requests.get(url=url, timeout=REQUEST_TIMEOUT_SECONDS) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.11/lib/python3.11/site-packages/requests/api.py\", line 73, in get return request(\"get\", url, params=params, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.11/lib/python3.11/site-packages/requests/api.py\", line 59, in request return session.request(method=method, url=url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.11/lib/python3.11/site-packages/requests/sessions.py\", line 575, in request prep = self.prepare_request(req) ^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.11/lib/python3.11/site-packages/requests/sessions.py\", line 484, in prepare_request p.prepare( File \"/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.11/lib/python3.11/site-packages/requests/models.py\", line 367, in prepare self.prepare_url(url, params) File \"/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.11/lib/python3.11/site-packages/requests/models.py\", line 438, in prepare_url raise MissingSchema( requests.exceptions.MissingSchema: Invalid URL '/.well-known/openid-configuration': No scheme supplied. Perhaps you meant https:///.well-known/openid-configuration? fetch_well_known failed ", "extra": {}, "correlation_id": "01JMNTYVDXZR2FC5ST20GAVGEW", "logger": "cloud_connector", "level": "error", "type": "mlops", "stage": "main", "timestamp": "2025-02-22T03:24:32.960558Z", "message": "Invalid URL '/.well-known/openid-configuration': No scheme supplied. Perhaps you meant https:///.well-known/openid-configuration?" } ``` ### Example Project <!--If possible, please create an example project here on GitLab.com that exhibits the problematic behavior, and link to it here in the bug report. If you are using an older version of GitLab, this will also determine whether the bug is fixed in a more recent version.--> ### What is the current _bug_ behavior? The `ai-gateway` configmap created by the Helm chart sets `AIGW_CUSTOMER_PORTAL_URL: ""` which is treated as an explicitly set value. ### What is the expected _correct_ behavior? An empty `AIGW_CUSTOMER_PORTAL_URL` value should be treated the same as the variable being undefined. ### Relevant logs and/or screenshots <!--Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's tough to read otherwise.--> ### Output of checks <!--If you are reporting a bug on GitLab.com, uncomment below--> <!--This bug happens on GitLab.com--> <!--and uncomment below if you have /label privileges--> <!--/label ~"reproduced on GitLab.com"--> <!--or follow up with an issue comment of `@gitlab-bot label ~"reproduced on GitLab.com"` if you do not--> #### Results of GitLab environment info <!--Input any relevant GitLab environment information if needed.--> <details> <summary>Expand for output related to GitLab environment info</summary> <pre> (For installations with omnibus-gitlab package run and paste the output of: \\\`sudo gitlab-rake gitlab:env:info\\\`) (For installations from source run and paste the output of: \\\`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production\\\`) </pre> </details> #### Results of GitLab application Check <!--Input any relevant GitLab application check information if needed.--> <details> <summary>Expand for output related to the GitLab application check</summary> <pre> (For installations with omnibus-gitlab package run and paste the output of: \`sudo gitlab-rake gitlab:check SANITIZE=true\`) (For installations from source run and paste the output of: \`sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true\`) (we will only investigate if the tests are passing) </pre> </details> ### Possible fixes <!--If you can, link to the line of code that might be responsible for the problem.--> <!--If you don't have /label privileges, follow up with an issue comment of `@gitlab-bot label ~"type::bug"`-->