Investigate 401 error when testing self-hosted model configuration

Problem

When attempting to test a newly created self-hosted model in a GitLab Dedicated instance, the test fails with a 401 error. The AI Gateway returns JWT authentication errors indicating "Not enough segments" in the token.

Related Issues

Error Logs from AI Gateway

JWT Parsing Error 

{
  "status_code": null,
  "exception_class": "JWTError",
  "backtrace": "Traceback (most recent call last):\n  File \"/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/jose/jws.py\", line 180, in _load\n    signing_input, crypto_segment = jwt.rsplit(b\".\", 1)\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nValueError: not enough values to unpack (expected 2, got 1)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/jose/jwt.py\", line 159, in decode\n    payload = jws.verify(token, key, algorithms, verify=verify_signature)\n              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/jose/jws.py\", line 74, in verify\n    header, payload, signing_input, signature = _load(token)\n                                                ^^^^^^^^^^^^\n  File \"/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/jose/jws.py\", line 184, in _load\n    raise JWSError(\"Not enough segments\")\njose.exceptions.JWSError: Not enough segments\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/gitlab_cloud_connector/providers.py\", line 116, in authenticate\n    jwt_claims = jwt.decode(\n                 ^^^^^^^^^^^\n  File \"/home/aigateway/app/venv/ai-gateway-M6hW6iiC-py3.12/lib/python3.12/site-packages/jose/jwt.py\", line 161, in decode\n    raise JWTError(e)\njose.exceptions.JWTError: Not enough segments\",
  "extra": {},
  "cc_python_version": "3.10.0",
  "correlation_id": "01KCW1FNWMZ1FQZVP8NDVB6BZ6",
  "logger": "cloud_connector",
  "level": "error",
  "type": "mlops",
  "stage": "main",
  "timestamp": "2025-12-19T19:30:45.539654Z",
  "message": "Not enough segments"
}

401 Response 

{
  "url": "[REDACTED]/v1/prompts/model_configuration/check",
  "path": "/v1/prompts/model_configuration/check",
  "status_code": 401,
  "method": "POST",
  "correlation_id": "01KCW1FNWMZ1FQZVP8NDVB6BZ6",
  "http_version": "1.1",
  "client_ip": "10.0.29.175",
  "client_port": 0,
  "duration_s": 0.0017395640024915338,
  "duration_request": 0.07598066329956055,
  "request_arrived_at": "2025-12-19T19:30:45.538225+00:00",
  "response_start_duration_s": 0.0016706850146874785,
  "first_chunk_duration_s": 0.0017097860109061003,
  "cpu_s": 0.001738841999952001,
  "content_type": "application/json",
  "user_agent": "Ruby",
  "gitlab_language_server_version": null,
  "gitlab_instance_id": "f798e820-207f-4849-869d-1f753edb3312",
  "gitlab_global_user_id": "pgkDzqQoXjWQxdwm4wEX1jID6Aa3OGvWDr4I9wLPqmw=",
  "gitlab_host_name": "[REDACTED]",
  "gitlab_version": "18.5.4",
  "gitlab_saas_duo_pro_namespace_ids": null,
  "gitlab_feature_enabled_by_namespace_ids": "",
  "gitlab_feature_enablement_type": "duo_enterprise",
  "gitlab_realm": "self-managed",
  "is_gitlab_team_member": "false",
  "enabled-instance-verbose-ai-logs": true,
  "auth_duration_s": 0.0015286520356312394,
  "token_issuer": "",
  "auth_error_details": "Forbidden by auth provider",
  "http_exception_details": "Forbidden by auth provider",
  "logger": "api.access",
  "level": "info",
  "type": "mlops",
  "stage": "main",
  "timestamp": "2025-12-19T19:30:45.540038Z",
  "message": "10.0.29.175:0 - \"POST /v1/prompts/model_configuration/check HTTP/1.1\" 401"
}

Context

  • The AI Gateway is deployed in an offline AWS environment
  • GitLab instance is using an offline license with Duo Enterprise addon
  • The gateway was configured with DUO_WORKFLOW_AUTH__OIDC_CUSTOMER_PORTAL_URL environment variable for offline license support
  • Chat functionality works correctly with the configured model, but the model configuration test endpoint fails

Troubleshooting Attempted

  • Upgraded AI Gateway Helm chart from 0.5.0 to 0.6.1
  • Added DUO_WORKFLOW_AUTH__OIDC_CUSTOMER_PORTAL_URL environment variable
  • Verified IAM user permissions
  • Disabled DUO_WORKFLOW_AUTH__ENABLED to test if auth was the root cause

Related Issue in AI Gateway Tracker

#400 (closed) - Similar JWT "not enough segments" errors reported