Backport of 'Authenticate self-hosted billing events with CustomersDot token' to 19.4
What does this MR do and why?
This MR backports !258285 (merged), "Authenticate self-hosted billing events with CustomersDot token", to 19-4-stable-ee for patch release 19.4.2. The original MR merged into master on 2026-10-02 (milestone 19.5) and is deployed to GitLab.com. The cherry-pick is clean. The touched files are identical on both branches before the fix.
Since GitLab 19.4, Self-Managed instances that use GitLab Duo Self-Hosted with an online cloud license get HTTP 403 "Forbidden by auth provider" on every Code Suggestions completion. Before each completion, GitLab sends a billing event to the cloud Duo Workflow Service. Since !250313 (merged), that call used the instance's self-signed token. The cloud service cannot verify this token and rejects the call. GitLab then blocks the completion before it reaches the customer's self-hosted AI Gateway. No workaround exists, so Code Suggestions is unusable for all users on affected instances. Only 19.4 is affected: !250313 (merged) is not in 19.3 or 19.2.
The billing call now sends the CustomersDot-issued instance JWT again. The shared AI Gateway header builder takes it as a bearer_token argument. If the JWT is missing, the billing call returns a clear error and does not fall back to a self-signed token. Calls to the self-hosted AI Gateway still use the self-signed token.
References
- Issue: #630581 (closed)
- Original MR: !258285 (merged)
- Regression introduced by: !250313 (merged)
- Request for help: https://gitlab.com/gitlab-com/request-for-help/-/work_items/5439
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
- This MR is backporting a bug fix, documentation update, or spec fix, previously merged in the default branch.
- The MR that fixed the bug on the default branch has been deployed to GitLab.com (not applicable for documentation or spec changes).
- The MR title is descriptive (e.g. "Backport of 'title of default branch MR'"). This is important, since the title will be copied to the patch blog post.
- Required labels have been applied to this merge request
- severity label and bug subtype labels (if applicable)
- If this MR fixes a bug that affects customers, the customer label has been applied.
- This MR has been approved by a maintainer (only one approval is required).
- Ensure the
E2E Omnibus GitLab EEpipeline (triggered by thee2e:test-on-omnibus-eejob) has succeeded. If it has failed (or shows warnings), investigate the failures. If you determine the failures are unrelated, you may proceed. If you need assistance investigating, request help in the #s_developer_experience Slack channel to confirm the failures are unrelated to the merge request.
Note to the merge request author and maintainer
If you have questions about the patch release process, please:
- Refer to the patch release runbook for engineers and maintainers for guidance.
- Ask questions on the
#releasesSlack channel (internal only). - Once the backport has been merged, the commit changes will be automatically deployed to a release environment that can be used for manual validation. See after merging runbook for details.