Docs: Fix AI Gateway upgrade tag and add the 19.6 upgrade note

Andrew Dunnrequested to merge
docs/aigw-upgrade-matching-tag into master

What does this MR do?

The upgrade section tells customers to download the newest AI Gateway image, while the install section says to use the tag that matches their GitLab version. A Docker customer on GitLab 19.5 or earlier can follow the upgrade section onto an AI Gateway 19.6 image, and some flows then fail to start. A start from the UI or the REST API fails with 400 Bad request and the gateway's message 16:Forbidden by auth provider. When GitLab starts the flow itself, the flow fails. For a code review on a merge request, GitLab adds a merge request note with Error code: DCR4005.

This MR:

  • Points the upgrade section at the matching tag.
  • Adds a troubleshooting topic for running AI Gateway 19.6 or later with GitLab 19.5 or earlier, and links it from the upgrade section.
  • Corrects the key rotation entry in offline_deployment.md, which named Could not obtain Duo Workflow token, a result the UI and REST API do not show. That line came from !260775 (merged).

It documents gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!7404 (merged), where @manojmj asked for it in gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!7404 (comment 4106050156). That MR merged on 3 November. The topic says 19.6 on the assumption that the 19.6 gateway release is cut after that merge.

The topic covers Docker deployments and flows. Helm chart and Docker Compose deployments are not changed here. Chat requests are fixed in GitLab 19.4.2, 19.3.4 and 19.2.8 (backports in !257901 (merged), !257903 (merged) and !257906 (merged)).

What happens when the gateway is newer than GitLab

sequenceDiagram
    participant GitLab as GitLab 19.5
    participant Gateway as AI Gateway 19.6
    GitLab->>Gateway: Flow request signed with the Customers Portal token
    alt Customers Portal URL not set
        Gateway-->>GitLab: UNAUTHENTICATED, Forbidden by auth provider
        Note over Gateway: logs Signature verification failed.
        Note over GitLab: UI and REST API: 400 Bad request, 16:Forbidden by auth provider<br/>Code review: MR note DCR4005
    else URL set to customers.gitlab.com, online license
        Gateway-->>GitLab: Request accepted
    end

Where each statement in the topic comes from:

How this was checked

Flow starts on a self-managed GitLab 19.5.3 instance, with an online license signed by the staging Customers Portal, went to an AI Gateway. Agents & flows used the GitLab AI vendor model and the flow used a self-hosted model. The gateways were CI images built with the self-hosted image's settings: the 7404 target branch before the change, and the 7404 merge result.

  • Customers Portal URL set to the staging portal, which signed the license: the flow starts on both builds. A start from the UI (the Implement button on an issue) and a start from the REST API (POST /api/v4/ai/duo_workflows/workflows with start_workflow) also succeed on the merge result.
  • URL not set, with 7404: the gateway does not load the Customers Portal keys and logs Signature verification failed..
    • A start from the UI shows 400 Bad request - ["16:Forbidden by auth provider. debug_error_string:{UNAUTHENTICATED:Forbidden by auth provider}"].
    • A start from the REST API returns HTTP 400 with the same message.
    • Running the code review service on a merge request returns Could not obtain Duo Workflow token and adds a note with Error code: DCR4005.
    • An empty value fails the same way on a start from a script.
  • URL set to https://customers.gitlab.com, with 7404: the gateway loads the Customers Portal keys, then rejects the token, because the test license is signed by the staging portal. This shows the variable takes effect. The customers.gitlab.com request went to the 7404 build only.

Not covered: a license signed by the production portal, either variable set alone, an offline license, and a code review that completes with the URL set. Before 7404, an unset URL also fails here. That gateway loads the production portal keys by default (7 key IDs, against 5 with 7404), and those keys cannot verify the test license. A pass before 7404 needs a license signed by the production portal.

Why the workaround holds for a production license. No test used a license signed by the production portal, so the pass is not observed. The source makes the two builds equivalent:

Related issues

Related to gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!7404 (merged)

Author's checklist

If you are a GitLab team member and only adding documentation, do not add any of the following labels:

  • ~"frontend"
  • ~"backend"
  • ~"type::bug"
  • ~"database"

These labels cause the MR to be added to code verification QA issues.

Reviewer's checklist

Documentation-related MRs should be reviewed by a Technical Writer for a non-blocking review, based on Documentation Guidelines and the Style Guide.

If you aren't sure which tech writer to ask, use roulette or ask in the #docs Slack channel.

  • If the content requires it, ensure the information is reviewed by a subject matter expert.
  • Technical writer review items:
    • Ensure docs metadata is present and up-to-date.
    • Ensure the appropriate labels are added to this MR.
    • Ensure a release milestone is set.
    • If relevant to this MR, ensure content topic type principles are in use, including:
      • The headings should be something you'd do a Google search for. Instead of Default behavior, say something like Default behavior when you close an issue.
      • The headings (other than the page title) should be active. Instead of Configuring GDK, say something like Configure GDK.
      • Any task steps should be written as a numbered list.
      • If the content still needs to be edited for topic types, you can create a follow-up issue with the docs-technical-debt label.
  • Review by assigned maintainer, who can always request/require the reviews above. Maintainer's review can occur before or after a technical writer review.

An AI coding assistant wrote this change. The assistant also checked each statement against the sources linked above, ran Vale and markdownlint-cli2 on the changed page, and ran the flow requests described above on a self-managed GitLab 19.5.3 instance on a test VM.

Edited by Andrew Dunn

Merge request reports