<!-- Bug template: https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Bug.md --> ### Summary On a GitLab Self-Managed instance with an **online cloud license** and **GitLab Duo Self-Hosted** (self-hosted AI Gateway + self-hosted model for Code Suggestions), every `POST /api/v4/code_suggestions/completions` returns **HTTP 403** since upgrading to **19.4.0** (also on 19.4.1 source, unchanged): ``` {"message":"16:Forbidden by auth provider. debug_error_string:{UNAUTHENTICATED:Forbidden by auth provider}"} ``` The request never reaches the self-hosted AI Gateway. The 403 is raised in `ee/lib/api/code_suggestions.rb:242-247` when `Ai::CodeSuggestions::SelfHostedBillingTracker#track` fails. The tracker opens a `TrackSelfHostedExecuteWorkflow` gRPC stream to GitLab's **cloud** Duo Workflow Service (`duo-workflow-svc.runway.gitlab.net:443`), but since !250313 it authenticates that stream with an **instance self-signed** Cloud Connector token instead of the CustomersDot-issued `ServiceAccessToken`. The cloud service has no key to validate an instance-signed token and answers `UNAUTHENTICATED`, so Code Suggestions are hard-blocked for every online-license Duo Self-Hosted customer. Same code path worked on 18.11.11 and 19.2.6 with the identical gateway, license, network and feature settings. ### Steps to reproduce 1. Self-Managed EE with an online cloud license (activation code, `licenses.cloud = true`, CustomersDot sync succeeding). 2. GitLab Duo Self-Hosted: self-hosted AI Gateway (`self-hosted-v19.4.1-ee`), a self-hosted model feature setting for Code Suggestions (`feature_setting.self_hosted? == true`). 3. Upgrade GitLab to 19.4.0. 4. `POST /api/v4/code_suggestions/completions` with any valid body (IDE or `curl`/`glab api`). ### What is the current *bug* behavior? HTTP 403 `16:Forbidden by auth provider … UNAUTHENTICATED` on every completion. Rails log for one request (correlation `01M36HKE29W5MJTQ57ZX9G88TQ`, 2026-09-23 07:10Z): ``` ee/lib/api/code_suggestions.rb:245 → ee/lib/ai/code_suggestions/self_hosted_billing_tracker.rb:25 track → ee/lib/ai/duo_workflow/duo_workflow_service/client.rb:53 track_self_hosted_client_event → GRPC::Unauthenticated "Cloud Connector key loaded cc_kid=pfdd5x…" # the instance signing key being loaded for this call ``` Nothing is logged on the self-hosted AI Gateway for these requests (only probes). The Duo health check in the Admin Area is fully green, including "The cloud-connected GitLab Duo Agent Platform service at duo-workflow-svc.runway.gitlab.net:443 is operational" and all billing prerequisites reachable. ### What is the expected *correct* behavior? Completions succeed (HTTP 200 from the self-hosted gateway) and the billing event is accepted by the cloud service, as on 19.2.6. If the billing call fails for a reason outside the customer's control, it should not hard-fail user-facing Code Suggestions with a 403 (the original tracker MR !225296 describes a 503). ### Root cause (verified against `v19.4.0-ee` and `master`) - `ee/lib/ai/code_suggestions/self_hosted_billing_tracker.rb`: `grpc_client` is built with `duo_workflow_service_url: ::Gitlab::DuoWorkflow::Client.cloud_connected_url(user:)` (the cloud DWS) **and** `feature_setting: feature_setting` (the self-hosted setting). - `ee/lib/ai/duo_workflow/duo_workflow_service/client.rb:190-206`: `metadata` = `Gitlab::AiGateway.headers(user:, unit_primitive_name: :duo_agent_platform, ai_feature_name: :duo_agent_platform, feature_setting:, …)`. - `ee/lib/gitlab/ai_gateway.rb:184-208` → `::CloudConnector::Tokens.get(unit_primitive:, resource: user, feature_setting:, …)`. - `ee/lib/cloud_connector/tokens.rb`: `use_self_signed_token?` returns true when `feature_setting&.self_hosted?` → `issue_token` signs a JWT with the instance's `cloud_connector_keys` private key (issuer = instance OIDC issuer). Otherwise it returns the CustomersDot-issued `ServiceAccessToken`. - Nothing publishes instance signing keys to CustomersDot or GitLab.com: `customers.gitlab.com/oauth/discovery/keys` serves 2 kids and `gitlab.com/oauth/discovery/keys` 4; none of our instance's kids appear in either. A GitLab-hosted backend therefore cannot validate the token → `UNAUTHENTICATED` from the DWS authentication interceptor (`Forbidden by auth provider`). - **Introduced by !250313** ("Fix Duo Agent Platform probe's token generation and duplicate gRPC calls", milestone 19.4). Its diff replaces `token: ::CloudConnector::Tokens.cloud_connector_token` with `feature_setting: feature_setting` in the tracker and removes the `token:` kwarg / `"Bearer #{token || cloud_connector_token}"` from the client. The MR description says the client "never read the token kwarg", but on 18.11.11 the billing call used `metadata` with that bearer (`client.rb:97`) and on 19.2.6 `generate_headers = generate_gateway_headers.merge(metadata)` so the explicit CustomersDot token still won (`client.rb:60-62,168`). It is the only commit touching the tracker between 19.2.6 and 19.4.0; 19.4.1 does not change it. - Within-instance discriminator: the Duo health check's `DuoAgentPlatformProbe` calls the same cloud service with `MockFeatureSetting.new(false)` → CustomersDot token → `ListTools` succeeds ("operational"), while the tracker passes the real self-hosted feature setting → instance-signed token → rejected. Same instance, same host, same minute; only the token type differs. (`ListTools` is not auth-exempt; only `grpc.health.v1.Health/Check|Watch` are.) - Not an entitlement/quota failure: those surface as `PERMISSION_DENIED` (`USAGE_QUOTA_NOT_ENTITLED`) or `RESOURCE_EXHAUSTED` in the DWS, not `UNAUTHENTICATED`. Same defect family as #621674 (token type chosen by the unit-primitive → feature-setting mapping; fixes !254156 / !256656 cover Duo Code Review, not the billing tracker). Related: !256332 (self-managed billing events must carry the CustomersDot-issued instance uid) shows the cloud billing path is meant to be authenticated with CustomersDot-issued material. ### Suggested fix The billing call targets a GitLab-hosted service, so it must use the CustomersDot `ServiceAccessToken` (as before !250313 and as the health-check probe does), while calls to the customer's own gateway keep the self-signed token. Note `tokens.rb` re-derives a feature setting from the unit primitive when none is passed, so simply dropping `feature_setting:` is not sufficient; the tracker/client need an explicit "cloud credential" path. Also consider not returning 403 to the end user when the billing stream fails. ### Workarounds evaluated (none acceptable) - No admin setting or env var disables `Ai::SelfHostedDapBilling.should_bill?` for online cloud licenses (only dev env, SaaS, or a non-online license exit early). - `CLOUD_CONNECTOR_SELF_SIGN_TOKENS` makes more calls self-signed (wrong direction). - Pointing `duo_workflow.service_url` at the self-hosted gateway is not renderable from the Helm chart and would silently swallow GitLab's metering. - An offline cloud license bypasses the gate but changes the licensing arrangement. ### Output of checks Not GitLab.com. #### Results of GitLab environment info <details> <summary>Expand</summary> <pre> GitLab 19.4.0-ee (revision fa703f93afa), Helm chart 10.4.0 on Kubernetes (EKS), online cloud license (Duo Enterprise). Self-hosted AI Gateway image registry.gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/model-gateway:self-hosted-v19.4.1-ee (2 replicas), gRPC Duo Workflow service on :50052. Egress from the Rails pods to customers.gitlab.com, cloud.gitlab.com and duo-workflow-svc.runway.gitlab.net:443 verified (TLS not intercepted). gitlab.yml has no duo_workflow: block; feature flag duo_workflow_cloud_connector_url is off. Same behaviour observed on 18.11.11 (200) → 19.2.6 (200) → 19.4.0 (403) with no other change. </pre> </details>