Pin the flow version a resumed Duo session runs on
What does this MR do and why?
A GitLab Duo Agent Platform session can run for many turns. For example, Duo Developer answers a mention, the user replies on the session page, and each reply starts a new CI job that resumes the same session. A session has to run one flow version for its whole life, and today it doesn't:
- Resumes fail after a flag flip. Every resume works out the flow version again from feature flags (
flow_version_resolver), but the CI job keeps the version from the first job. Since !260255 (merged) the workflow token is bound to the flow version, so if a flag changes between two turns, the token and the job name different versions and the Duo Workflow Service rejects the resume withPERMISSION_DENIED. - Sessions switch version silently. With an unchanged constraint such as
^4.0.0, a newly shipped4.1.0takes over an open session in the middle of a conversation.
This blocks continuable Duo Developer sessions (gitlab-org#24022): its flag duo_developer_continuable_sessions selects developer/^4.0.0, and turning it off would break every open session.
The rule: Rails writes down which flow a session was allowed to start with, the first checkpoint may only make that more exact, and every later run uses that record.
What changes:
- When a session is created, Rails records the flow it starts in the existing
flow_metadata_jsonfield and marks the record withflow_version_constraint. For example:{"flow_id":"developer","schema_version":"v1","flow_version":"^4.0.0","flow_version_constraint":"^4.0.0"}. - The first checkpoint (the gateway reports the exact version it runs, through the executor) may only narrow
flow_version: same flow ID, same schema, and a version that satisfies the constraint, for example4.0.0. Anything else leaves the record as Rails wrote it. Later checkpoints don't touch it. - Every run of the session reads one method,
Workflow#foundational_flow_params, which returns the record. The token and the CI job variables of a resume both come from it, so they always name the same version.
Without the marker, nothing changes: sessions created before this MR, Duo CLI sessions, and custom AI Catalog flows resolve as they do today, and a checkpoint stores the client's report as sent. There is no new column and no migration.
Details: what each changed file does
The primitive
Ai::DuoWorkflows::FlowMetadataowns the record:startedbuilds it,narroweddecides what the first checkpoint may store,pinned_paramsturns it back into start params. Malformed JSON or non-string values count as "no record", so the session resolves as today.CreateWorkflowService#started_flow_metadatawrites the record once at creation, from a trustedstarted_flow:keyword or elseFoundationalFlowStartParamsResolver. Callers can't setflow_metadata_json: it's removed from the incoming params.CreateCheckpointServicepasses the first checkpoint'sflow_metadata_jsonthroughFlowMetadata.narrowed.Ai::DuoWorkflows::FlowVersionConstraintimplements the part of the gateway's version constraints that flows use: an exact match, or^X.Y.Zfor stable versions. Anything else counts as unmet, so the record keeps the constraint. There is no Rails prior art:semver_dialectscan't parse^or~, andAi::Catalog::Item#resolve_versionmatches exact versions only.Ai::Catalog::ExecuteWorkflowServicepasses the flow config a server-side caller names asstarted_flow:.
Callers that now read the session instead of resolving again
| Path | Before |
|---|---|
Runtimes::Ci (CI start and resume through ExecuteRunService) |
token minted from a fresh resolve |
ResumeWorkplanService |
fresh resolve |
POST …/workflows/:id/resume, foundational flows (workflow_credential_params(workflow:)) |
fresh resolve |
POST …/workflows/:id/resume, catalog flows (Flows::ExecuteService#resolved_foundational_flow_params) |
fresh resolve |
ServerSideExecutionService (Workhorse request body, for example Slack) |
fresh resolve |
ConnectionConfigService#trusted_flow_config_id (models reserved for trusted flows; only bl_security sets gitlab_hosted_runner_only) |
fresh resolve |
ResumeWorkflowService#variables |
kept the first job's flow variables; now takes them from the params the token was minted from |
Details: behaviour changes and what this doesn't cover
- No feature flag. The result differs from today only for resumes that fail or would switch version, and for the mention assistant below.
- The
@GitLabDuomention assistant now resumes on its own flow. It runsgitlab_duo_mention_assistantunder thecode_review/v1definition. Its record now holds the flow config the caller names (gitlab_duo_mention_assistant/v1), so a resume after a tool approval mints the token and job for the mention assistant, notcode_review. - A requested Code Review depth still picks the flow.
CreateAndStartWorkflowServiceresolves the flow with the request'sadditional_contextand records the result, so astandardordeepreview is pinned to the flow that depth selects (code_reviewordeep_code_review). GitlabDuoNote::FLOW_VERSIONchanges from'v1'to'^1.0.0'(ee/app/services/ai/messaging/adapters/gitlab_duo_note.rb). The gateway readv1as exactly1.0.0, but the pin can't narrowv1, soflowMetadataVersionwould stay atv1for the whole session. With^1.0.0the first checkpoint narrows it to1.0.0, and new mention-assistant sessions start on the latest1.x.flowMetadataVersionin GraphQL shows the constraint before the first checkpoint, for example^4.0.0, and the exact version after it. No code change in the presenter.- A fix in a newer minor no longer reaches open sessions. This is intended (decided with the chat engine owners in gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!7419 (merged)). A registry version therefore has to stay available while sessions use it.
- Web chat and the Duo CLI build their own start request, so Rails has no record for them and keeps storing what the client reports.
- Plain
chatsessions didn't record aflow_idin local testing, so they keep resolving. That's safe, but they still pick up new minors until the chat engine cutover. - The gateway half (routing chat sessions by the pin at the engine cutover) is out of scope.
References
- Closes #630374 (closed)
- Continued Duo Developer sessions: gitlab-org#24022
- Token bound to the flow version: !260255 (merged)
- ADR-001, sessions rest between turns and keep their version: gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!7419 (merged)
- Flag that selects
developer/^4.0.0: !262809 (merged) - Related, consolidating how flows start: #629005
Screenshots or screen recordings
Not applicable: backend only.
How to set up and validate locally
This walkthrough covers three cases: a Duo Developer session that keeps its version after the flag that selected it is turned off (example 1), the @GitLabDuo mention assistant recording its own flow (example 3), and a first checkpoint that tries to claim a version Rails never allowed (example 4).
developer/4.0.0 (what the flag selects) can't run a turn yet, because it needs the chat engine. Step 2 therefore temporarily points the flag at developer/2.1.0-interactive, which pauses at input_required after each answer and can be resumed the same way.
Before you start
- GDK with the Duo Workflow Service running from the gateway's
mainbranch (it hasdeveloper/2.1.0-interactive). - A GDK runner that picks up Duo Agent Platform jobs (Docker running).
- A project where the Developer flow is turned on, its service account (for example
duo-developer-<group>) is a member, and you have an issue and an MR to comment on. - A personal access token with the
apiscope.
Steps
-
Check out this branch:
git fetch origin duo/fix/630374-pin-flow-version-on-resume git checkout duo/fix/630374-pin-flow-version-on-resume -
Temporarily point the flag at a version that can be resumed, in
ee/app/models/ai/catalog/foundational_flow/developer/definition.rb:- [REFERENCE, '^4.0.0'] + [REFERENCE, '2.1.0-interactive']Then restart Rails:
gdk restart rails-web rails-background-jobs. -
In
bin/rails console, define this helper. It prints the record, what the next run uses, and for each job the version in its variables next to the version in its token:def flow_versions(id) w = Ai::DuoWorkflows::Workflow.find(id) puts "record: #{w.flow_metadata_json}" puts "resumes with: #{w.foundational_flow_params.inspect}" w.workloads.order(:id).each do |workload| vars = workload.pipeline.builds.first.variables.to_hash payload = vars['DUO_WORKFLOW_SERVICE_TOKEN'].split('.')[1] claims = Gitlab::Json.parse(Base64.urlsafe_decode64(payload + ('=' * (-payload.size % 4)))) puts "job #{workload.id}: DUO_WORKFLOW_FLOW_VERSION=#{vars['DUO_WORKFLOW_FLOW_VERSION']} " \ "token flow_version=#{claims['flow_version']}" end end
Example 1: the version survives a flag flip
-
Turn the flag on for your user:
bin/rails runner 'Feature.enable(:duo_developer_continuable_sessions, User.find_by_username("<you>"))' -
Comment on the issue:
@<developer service account> In one sentence: which port do the servers listen on? Don't change files.Expect: within 1–3 minutes the service account replies, and the session (Automate > Sessions, newest entry) shows Input required.
flow_versions(<id>)prints the record{"flow_id":"developer","schema_version":"v1","flow_version":"2.1.0-interactive","flow_version_constraint":"2.1.0-interactive"}and one job where both versions are2.1.0-interactive. -
Turn the flag off:
bin/rails runner 'Feature.disable(:duo_developer_continuable_sessions, User.find_by_username("<you>"))'Expect: a new session would now get
^3.0.0, butflow_versions(<id>)still printsresumes with: {… flow_version: "2.1.0-interactive"}. -
Resume the session:
curl --request POST --header "PRIVATE-TOKEN: <token>" \ --data "human_approval=false" --data-urlencode "human_message=Thanks. Which file is the entry point?" \ "http://gdk.test:3000/api/v4/ai/duo_workflows/workflows/<id>/resume"Expect: HTTP
201. Within 1–3 minutes the session answers and is back at Input required.flow_versions(<id>)shows a second job withDUO_WORKFLOW_FLOW_VERSION=2.1.0-interactive token flow_version=2.1.0-interactive.On
masterthe same resume showstoken flow_version=^3.0.0next toDUO_WORKFLOW_FLOW_VERSION=2.1.0-interactive. If your Duo Workflow Service enforces authentication, that job fails withWorkflow token is not authorized for the requested flow version.
Example 3: the mention assistant records its own flow
-
Comment on an MR in the project:
@GitLabDuo In one sentence, what does this MR change?Expect: GitLab Duo replies on the MR. The session's definition is
code_review/v1, butflow_versions(<id>)prints the record{"flow_id":"gitlab_duo_mention_assistant","schema_version":"v1","flow_version":"1.0.0","flow_version_constraint":"^1.0.0"}("flow_version":"^1.0.0"before the first checkpoint) andresumes with: {flow_config_id: "gitlab_duo_mention_assistant", …}. The job and its token namegitlab_duo_mention_assistant. If the session pauses for a tool approval, the resumed job names it too, notcode_review.
Example 4: a first checkpoint can narrow the version, not replace it
With the flag still off, run in bin/rails console:
user = User.find_by_username('<you>')
project = Project.find_by_full_path('<your project>')
def new_session(user, project)
Ai::DuoWorkflows::CreateWorkflowService.new(container: project, current_user: user,
params: { workflow_definition: 'developer/v1', goal: 'test', environment: 'web' }).execute[:workflow]
end
def first_checkpoint(workflow, version)
Ai::DuoWorkflows::CreateCheckpointService.new(workflow: workflow, params: {
thread_ts: Gitlab::Utils.uuid_v7, checkpoint: { key: 'value' }, metadata: { step: 1 },
flow_metadata_json: { flow_id: 'developer', schema_version: 'v1', flow_version: version }.to_json
}).execute
workflow.reload.flow_metadata_json
end
forged = new_session(user, project)
first_checkpoint(forged, '9.0.0-dev')
fitting = new_session(user, project)
first_checkpoint(fitting, '3.0.0')Expect:
- Both sessions start with the record
{"flow_id":"developer","schema_version":"v1","flow_version":"^3.0.0","flow_version_constraint":"^3.0.0"}. forged: the checkpoint succeeds, the record is unchanged, andforged.foundational_flow_paramsreturnsflow_version: "^3.0.0", not9.0.0-dev.fitting: the record narrows to"flow_version":"3.0.0"and keeps the constraint. A second checkpoint doesn't change it.
Delete both sessions afterwards: [forged, fitting].each(&:destroy!).
Clean up
Revert the change from step 2 and restart Rails.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.