Pin the flow version a resumed Duo session runs on

What does this MR do and why?

A GitLab Duo Agent Platform session can run for many turns. For example, Duo Developer answers a mention, the user replies on the session page, and each reply starts a new CI job that resumes the same session. A session has to run one flow version for its whole life, and today it doesn't:

  • Resumes fail after a flag flip. Every resume works out the flow version again from feature flags (flow_version_resolver), but the CI job keeps the version from the first job. Since !260255 (merged) the workflow token is bound to the flow version, so if a flag changes between two turns, the token and the job name different versions and the Duo Workflow Service rejects the resume with PERMISSION_DENIED.
  • Sessions switch version silently. With an unchanged constraint such as ^4.0.0, a newly shipped 4.1.0 takes over an open session in the middle of a conversation.

This blocks continuable Duo Developer sessions (gitlab-org#24022): its flag duo_developer_continuable_sessions selects developer/^4.0.0, and turning it off would break every open session.

The rule: Rails writes down which flow a session was allowed to start with, the first checkpoint may only make that more exact, and every later run uses that record.

What changes:

  1. When a session is created, Rails records the flow it starts in the existing flow_metadata_json field and marks the record with flow_version_constraint. For example: {"flow_id":"developer","schema_version":"v1","flow_version":"^4.0.0","flow_version_constraint":"^4.0.0"}.
  2. The first checkpoint (the gateway reports the exact version it runs, through the executor) may only narrow flow_version: same flow ID, same schema, and a version that satisfies the constraint, for example 4.0.0. Anything else leaves the record as Rails wrote it. Later checkpoints don't touch it.
  3. Every run of the session reads one method, Workflow#foundational_flow_params, which returns the record. The token and the CI job variables of a resume both come from it, so they always name the same version.

Without the marker, nothing changes: sessions created before this MR, Duo CLI sessions, and custom AI Catalog flows resolve as they do today, and a checkpoint stores the client's report as sent. There is no new column and no migration.

Details: what each changed file does

The primitive

  • Ai::DuoWorkflows::FlowMetadata owns the record: started builds it, narrowed decides what the first checkpoint may store, pinned_params turns it back into start params. Malformed JSON or non-string values count as "no record", so the session resolves as today.
  • CreateWorkflowService#started_flow_metadata writes the record once at creation, from a trusted started_flow: keyword or else FoundationalFlowStartParamsResolver. Callers can't set flow_metadata_json: it's removed from the incoming params.
  • CreateCheckpointService passes the first checkpoint's flow_metadata_json through FlowMetadata.narrowed.
  • Ai::DuoWorkflows::FlowVersionConstraint implements the part of the gateway's version constraints that flows use: an exact match, or ^X.Y.Z for stable versions. Anything else counts as unmet, so the record keeps the constraint. There is no Rails prior art: semver_dialects can't parse ^ or ~, and Ai::Catalog::Item#resolve_version matches exact versions only.
  • Ai::Catalog::ExecuteWorkflowService passes the flow config a server-side caller names as started_flow:.

Callers that now read the session instead of resolving again

Path Before
Runtimes::Ci (CI start and resume through ExecuteRunService) token minted from a fresh resolve
ResumeWorkplanService fresh resolve
POST …/workflows/:id/resume, foundational flows (workflow_credential_params(workflow:)) fresh resolve
POST …/workflows/:id/resume, catalog flows (Flows::ExecuteService#resolved_foundational_flow_params) fresh resolve
ServerSideExecutionService (Workhorse request body, for example Slack) fresh resolve
ConnectionConfigService#trusted_flow_config_id (models reserved for trusted flows; only bl_security sets gitlab_hosted_runner_only) fresh resolve
ResumeWorkflowService#variables kept the first job's flow variables; now takes them from the params the token was minted from
Details: behaviour changes and what this doesn't cover
  • No feature flag. The result differs from today only for resumes that fail or would switch version, and for the mention assistant below.
  • The @GitLabDuo mention assistant now resumes on its own flow. It runs gitlab_duo_mention_assistant under the code_review/v1 definition. Its record now holds the flow config the caller names (gitlab_duo_mention_assistant / v1), so a resume after a tool approval mints the token and job for the mention assistant, not code_review.
  • A requested Code Review depth still picks the flow. CreateAndStartWorkflowService resolves the flow with the request's additional_context and records the result, so a standard or deep review is pinned to the flow that depth selects (code_review or deep_code_review).
  • GitlabDuoNote::FLOW_VERSION changes from 'v1' to '^1.0.0' (ee/app/services/ai/messaging/adapters/gitlab_duo_note.rb). The gateway read v1 as exactly 1.0.0, but the pin can't narrow v1, so flowMetadataVersion would stay at v1 for the whole session. With ^1.0.0 the first checkpoint narrows it to 1.0.0, and new mention-assistant sessions start on the latest 1.x.
  • flowMetadataVersion in GraphQL shows the constraint before the first checkpoint, for example ^4.0.0, and the exact version after it. No code change in the presenter.
  • A fix in a newer minor no longer reaches open sessions. This is intended (decided with the chat engine owners in gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!7419 (merged)). A registry version therefore has to stay available while sessions use it.
  • Web chat and the Duo CLI build their own start request, so Rails has no record for them and keeps storing what the client reports.
  • Plain chat sessions didn't record a flow_id in local testing, so they keep resolving. That's safe, but they still pick up new minors until the chat engine cutover.
  • The gateway half (routing chat sessions by the pin at the engine cutover) is out of scope.

References

Screenshots or screen recordings

Not applicable: backend only.

How to set up and validate locally

This walkthrough covers three cases: a Duo Developer session that keeps its version after the flag that selected it is turned off (example 1), the @GitLabDuo mention assistant recording its own flow (example 3), and a first checkpoint that tries to claim a version Rails never allowed (example 4).

developer/4.0.0 (what the flag selects) can't run a turn yet, because it needs the chat engine. Step 2 therefore temporarily points the flag at developer/2.1.0-interactive, which pauses at input_required after each answer and can be resumed the same way.

Before you start

  • GDK with the Duo Workflow Service running from the gateway's main branch (it has developer/2.1.0-interactive).
  • A GDK runner that picks up Duo Agent Platform jobs (Docker running).
  • A project where the Developer flow is turned on, its service account (for example duo-developer-<group>) is a member, and you have an issue and an MR to comment on.
  • A personal access token with the api scope.

Steps

  1. Check out this branch:

    git fetch origin duo/fix/630374-pin-flow-version-on-resume
    git checkout duo/fix/630374-pin-flow-version-on-resume
  2. Temporarily point the flag at a version that can be resumed, in ee/app/models/ai/catalog/foundational_flow/developer/definition.rb:

    -                [REFERENCE, '^4.0.0']
    +                [REFERENCE, '2.1.0-interactive']

    Then restart Rails: gdk restart rails-web rails-background-jobs.

  3. In bin/rails console, define this helper. It prints the record, what the next run uses, and for each job the version in its variables next to the version in its token:

    def flow_versions(id)
      w = Ai::DuoWorkflows::Workflow.find(id)
      puts "record:        #{w.flow_metadata_json}"
      puts "resumes with:  #{w.foundational_flow_params.inspect}"
      w.workloads.order(:id).each do |workload|
        vars = workload.pipeline.builds.first.variables.to_hash
        payload = vars['DUO_WORKFLOW_SERVICE_TOKEN'].split('.')[1]
        claims = Gitlab::Json.parse(Base64.urlsafe_decode64(payload + ('=' * (-payload.size % 4))))
        puts "job #{workload.id}: DUO_WORKFLOW_FLOW_VERSION=#{vars['DUO_WORKFLOW_FLOW_VERSION']}  " \
             "token flow_version=#{claims['flow_version']}"
      end
    end

Example 1: the version survives a flag flip

  1. Turn the flag on for your user:

    bin/rails runner 'Feature.enable(:duo_developer_continuable_sessions, User.find_by_username("<you>"))'
  2. Comment on the issue: @<developer service account> In one sentence: which port do the servers listen on? Don't change files.

    Expect: within 1–3 minutes the service account replies, and the session (Automate > Sessions, newest entry) shows Input required. flow_versions(<id>) prints the record {"flow_id":"developer","schema_version":"v1","flow_version":"2.1.0-interactive","flow_version_constraint":"2.1.0-interactive"} and one job where both versions are 2.1.0-interactive.

  3. Turn the flag off:

    bin/rails runner 'Feature.disable(:duo_developer_continuable_sessions, User.find_by_username("<you>"))'

    Expect: a new session would now get ^3.0.0, but flow_versions(<id>) still prints resumes with: {… flow_version: "2.1.0-interactive"}.

  4. Resume the session:

    curl --request POST --header "PRIVATE-TOKEN: <token>" \
      --data "human_approval=false" --data-urlencode "human_message=Thanks. Which file is the entry point?" \
      "http://gdk.test:3000/api/v4/ai/duo_workflows/workflows/<id>/resume"

    Expect: HTTP 201. Within 1–3 minutes the session answers and is back at Input required. flow_versions(<id>) shows a second job with DUO_WORKFLOW_FLOW_VERSION=2.1.0-interactive token flow_version=2.1.0-interactive.

    On master the same resume shows token flow_version=^3.0.0 next to DUO_WORKFLOW_FLOW_VERSION=2.1.0-interactive. If your Duo Workflow Service enforces authentication, that job fails with Workflow token is not authorized for the requested flow version.

Example 3: the mention assistant records its own flow

  1. Comment on an MR in the project: @GitLabDuo In one sentence, what does this MR change?

    Expect: GitLab Duo replies on the MR. The session's definition is code_review/v1, but flow_versions(<id>) prints the record {"flow_id":"gitlab_duo_mention_assistant","schema_version":"v1","flow_version":"1.0.0","flow_version_constraint":"^1.0.0"} ("flow_version":"^1.0.0" before the first checkpoint) and resumes with: {flow_config_id: "gitlab_duo_mention_assistant", …}. The job and its token name gitlab_duo_mention_assistant. If the session pauses for a tool approval, the resumed job names it too, not code_review.

Example 4: a first checkpoint can narrow the version, not replace it

With the flag still off, run in bin/rails console:

user = User.find_by_username('<you>')
project = Project.find_by_full_path('<your project>')

def new_session(user, project)
  Ai::DuoWorkflows::CreateWorkflowService.new(container: project, current_user: user,
    params: { workflow_definition: 'developer/v1', goal: 'test', environment: 'web' }).execute[:workflow]
end

def first_checkpoint(workflow, version)
  Ai::DuoWorkflows::CreateCheckpointService.new(workflow: workflow, params: {
    thread_ts: Gitlab::Utils.uuid_v7, checkpoint: { key: 'value' }, metadata: { step: 1 },
    flow_metadata_json: { flow_id: 'developer', schema_version: 'v1', flow_version: version }.to_json
  }).execute
  workflow.reload.flow_metadata_json
end

forged = new_session(user, project)
first_checkpoint(forged, '9.0.0-dev')

fitting = new_session(user, project)
first_checkpoint(fitting, '3.0.0')

Expect:

  • Both sessions start with the record {"flow_id":"developer","schema_version":"v1","flow_version":"^3.0.0","flow_version_constraint":"^3.0.0"}.
  • forged: the checkpoint succeeds, the record is unchanged, and forged.foundational_flow_params returns flow_version: "^3.0.0", not 9.0.0-dev.
  • fitting: the record narrows to "flow_version":"3.0.0" and keeps the constraint. A second checkpoint doesn't change it.

Delete both sessions afterwards: [forged, fitting].each(&:destroy!).

Clean up

Revert the change from step 2 and restart Rails.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports

Loading
Loading