Start fix_pipeline from MCP on the pipeline's branch and merge request

Fred de Gierrequested to merge
fdegier/631689-start-duo-session-mr-context into master

What does this MR do and why?

When fix_pipeline/v1 is started through the API with the failed pipeline's URL as its goal and no branch, the session now runs on that pipeline's branch, and the flow is told which merge request the pipeline belongs to. Before, it ran on the default branch with no merge request, so on GitLab.com it diagnosed the failure and stopped. Closes #631689 (closed).

The workflows API route works this out from the goal, so start_duo_session and any other REST caller get it. It sends the same context the automatic pipeline-failure trigger sends, built in one shared place. code_review/v1 gets its merge request's source branch the same way. The UI already sends the branch and context itself, so nothing changes for it: the route only fills them in when the caller sends no branch.

Why the goal, and why the route

The issue preferred adding a merge_request_url or pipeline_url parameter to the MCP tool (its option B). For fix_pipeline/v1 the goal already is that pipeline URL, so a parameter would ask for something the caller has already given, and the two could disagree. With the goal as the only input, the exact call that failed in the walkthrough now works unchanged.

The first version of this MR did the derivation in the MCP tool. In review, @terrichu pointed out that this left other REST callers broken, resolved the goal in a third place, and needed the current user in every aggregated MCP tool. It now lives in the route, next to the goal validation, and the MCP tool only gains a better description of goal.

Three places where this departs from the issue or the review:

  • A pipeline with no open merge request is not an error. The session runs on the pipeline's branch with an empty merge request URL, and the flow opens a new merge request with the fix. That is what the pipeline-failure trigger and the pipeline page's button already do.
  • A goal that is not a pipeline in the project is refused for MCP callers only. A goal_validator_resolver on fix_pipeline/v1 would also run for UI starts and triggers, and the merge request widget's button sends the target project with the pipeline's URL, which for a fork merge request is in the fork. REST callers keep the contract they had: the session starts, without the context.
  • developer/v1 on an existing merge request is not covered. The UI's "Resolve with Duo" buttons send the merge request's branch, but there the goal is prose. No journey in &21395 (closed) needs it yet.
For agents

Where it happens. goal_resource_params in the helpers block of ee/lib/api/ai/duo_workflows/workflows.rb, called right after validate_flow_goal! on both paths of POST /ai/duo_workflows/workflows, before any workflow is created:

  • the workflow_definition path merges its result into start_workflow_params;
  • the ai_catalog_item_consumer_id path feeds it into flow_params ahead of build_additional_context, which on its own only runs for consumers.

It does nothing when the caller sent a source_branch, so the UI's requests are untouched. A caller that sends some context but no branch keeps the categories it sent, and only the missing ones are added.

What the goal names comes from the flow definitions. A flow with a source_pipeline_resolver (only fix_pipeline/v1) gets the pipeline's source_ref and the merge_request and pipeline envelopes. A flow whose noteable_resolver returns a merge request, such as code_review/v1, gets that merge request's source_branch, which is what ReviewMergeRequestService sends.

Authorization. The route has already checked the caller can access the project and create a workflow. On top of that, the pipeline must pass read_pipeline, the merge request named in the context must pass read_merge_request (otherwise its URL is sent empty), and a code_review/v1 merge request must pass read_merge_request before its branch is used.

The shared builder. Ai::DuoWorkflows::AdditionalContext::PipelineContextBuilder is RunService#pipeline_additional_context moved out unchanged, plus a merge_request: keyword so the route can pass the merge request it has checked. These are the legacy merge_request and pipeline categories that fix_pipeline 1.0.4 reads (context:inputs.merge_request.url, context:inputs.pipeline.source_branch).

Docs. The source_branch attribute in doc/api/duo_agent_platform_flows.md and in the route's param description now says what happens when it is not specified. openapi_v3.yaml is regenerated, which also renames the request body schema, because its name hashes the content.

Still version 0.1.0. Only the goal description of start_duo_session changes, and the tool has not shipped in a release yet. The full-schema lock from !258826 (merged) is updated to it.

Specs. The REST behaviour is in workflows_spec.rb under 'when the goal names what a foundational flow runs against', and the MCP path and its refusal are in ee/spec/requests/api/mcp/handlers/call_tool_spec.rb. Removing the read_merge_request check or the MCP refusal fails 'when the caller cannot read the merge request the pipeline belongs to' and 'refuses without starting a session, and says what the goal must be'. They ran locally against 0a80f055d, the base the local bundle matches. The rebased branch needs gitlab-labkit 5.3.0, which that bundle does not have yet, so the run on the rebased code is this pipeline's.

One shared example is untouched on purpose. 'foundational flow workflow creation' starts fix_pipeline/v1 with the prose goal "Fix the pipeline" and expects 201. It still passes, which is the REST contract the second point above keeps.

consumer.item without safe navigation. ai_catalog_item_consumers.ai_catalog_item_id is NOT NULL with an ON DELETE RESTRICT foreign key, and the association is optional: false, so a consumer always has its item. The &. only left a branch rspec:undercoverage could never see covered.

Overlap with source_branch_resolver. Since b1f71a1e, a foundational flow can declare a source_branch_resolver, which start_workflow_params and Flows::ExecuteService fall back to when the caller sends no branch. fix_pipeline/v1 and code_review/v1 do not declare one, so for them the branch still comes from goal_resource_params. Moving their branch onto that hook would leave the route with only the context and the MCP refusal.

References

Closes #631689 (closed). Related: !254979 (merged) (start_duo_session), !254746 (merged), &21395 (closed) (walkthrough, step 1.10).

Screenshots or screen recordings

No UI change. Below is what a session is started with, from the same calls against master and against this branch on a local GDK. The target is a merge request whose pipeline fails on a shell syntax error.

Call master This branch
start_duo_session, pipeline URL as goal main, no context (session 1019) the MR branch, with the MR and pipeline context (session 1020)
REST, pipeline URL as goal, no branch main, no context (session 1031) the MR branch, with the MR and pipeline context (session 1032)
REST, branch and context sent, as the UI does — exactly what was sent (session 1033)
start_duo_session, another project's pipeline starts on main, no context (session 1036) refused, nothing started
REST, another project's pipeline starts on main, no context (session 1037) starts on main, as before (session 1034)

On master, the agent in this GDK still found the merge request without being given it, from the session's resource context, and posted a suggestion. On GitLab.com on 25 September, the same flow gave up instead ("Since the merge request URL is None … I cannot post diff notes"). So on master the outcome depends on the agent working it out for itself. On this branch the flow gets the merge request and the branch every time.

How to set up and validate locally

Steps, with what I got

This needs duo-workflow-service, gitlab-ai-gateway and a registered runner, because the point is a real fix_pipeline run. I served this branch from a worktree on a spare port, against the same database, with a config/puma.rb of workers 0 because the start call makes gRPC calls: RAILS_ENV=development bin/rails server -p 3021 -b 127.0.0.1. Port 3000 kept serving master. The served code is this branch's diff applied to 0a80f055d, because the rebased branch needs gitlab-labkit 5.3.0, which my local bundle does not have yet.

The Ruby below was run with bin/rails runner from the main checkout. It works the same pasted into rails console.

  1. Make two tokens, one with the mcp scope and one with api. I wrote mine to files instead of printing them:

    PersonalAccessToken.create!(
  user: User.find_by_username('root'), name: 'mcp-631689', scopes: [:mcp],
  expires_at: 7.days.from_now,
  organization: Organizations::Organization.default_organization
).token
    export MCP='glpat-...'   # the mcp-scoped token
export API='glpat-...'   # the same call with scopes: [:api]

  2. Make a merge request whose pipeline fails, in a project with Fix CI/CD Pipeline enabled. gitlab-duo/test has it, and its CI runs on merge request events:

    project = Project.find_by_full_path('gitlab-duo/test')
user = User.find_by_username('root')
branch = 'mcp-fix-pipeline-631689'
ci = project.repository.blob_at('main', '.gitlab-ci.yml').data
anchor = "    - echo \"Compiling source code\"\n"
raise 'anchor missing' unless ci.include?(anchor)
ci = ci.sub(anchor, anchor + "    - sh scripts/greet.sh\n")
script = "#!/bin/sh\necho \"Hello from $CI_PROJECT_NAME\n"
res = Files::MultiService.new(project, user,
  start_branch: 'main', branch_name: branch, commit_message: 'Greet from the build job',
  actions: [
    { action: :create, file_path: 'scripts/greet.sh', content: script },
    { action: :update, file_path: '.gitlab-ci.yml', content: ci }
  ]).execute
mr = MergeRequests::CreateService.new(project: project, current_user: user,
  params: { source_branch: branch, target_branch: 'main', title: 'Greet from the build job' }).execute

    What I got: !172. Two things went wrong for me here that you may hit as well:

    • No pipeline appeared. GDK's Sidekiq had been up for nine days, with gems older than the checkout, so NewMergeRequestWorker failed with undefined method `follow=' for an instance of Gitaly::ListCommitsRequest. Restart it with gdk restart rails-background-jobs. I also created this one pipeline by hand: MergeRequests::CreatePipelineService.new(project: project, current_user: user).execute(mr).

    • The job died before running its script, with fatal: missing blob object during a depth-20 fetch. The object is on the server, and workload jobs, which fetch the full history, check out fine. I added GIT_DEPTH: "0" to the branch's variables, and the next pipeline (1625) failed for the intended reason:

      $ sh scripts/greet.sh
scripts/greet.sh: 3: Syntax error: Unterminated quoted string

  3. Every session below was inspected with this script, which reads the CI job's variables, where the branch and the context end up. Run it as bin/rails runner inspect_session.rb 1032 stop; stop cancels the session afterwards so the runner is free for the next one:

    inspect_session.rb 
    id = ARGV.fetch(0).to_i
w = Ai::DuoWorkflows::Workflow.find(id)
mr = w.project.merge_requests.find_by(iid: 172)
puts "session #{w.id} status=#{w.status_name} definition=#{w.workflow_definition} source_type=#{w.source_type}"
puts "linked merge request: !#{w.merge_request&.iid.inspect}  source pipeline: #{w.pipeline_links.link_type_source.first&.pipeline_id.inspect}"
workload = w.workloads.last
build = workload&.pipeline&.builds&.first
unless build
  puts "no workload yet"
  exit
end
vars = build.yaml_variables.to_h { |v| [v[:key], v[:value]] }
puts "DUO_WORKFLOW_SOURCE_BRANCH=#{vars['DUO_WORKFLOW_SOURCE_BRANCH']}"
puts "workload checks out #{workload.pipeline.sha[0, 8]} (MR !172 head #{mr.diff_head_sha[0, 8]}, main #{w.project.commit('main').sha[0, 8]})"
context = Gitlab::Json.safe_parse(vars['DUO_WORKFLOW_ADDITIONAL_CONTEXT_CONTENT'].to_s) || []
context.each do |envelope|
  next unless %w[merge_request pipeline].include?(envelope['Category'])

  puts "context #{envelope['Category']}: #{envelope['Content']}"
end
puts "no merge_request or pipeline context" if context.none? { |e| %w[merge_request pipeline].include?(e['Category']) }
if ARGV[1] == 'stop'
  pl = workload.pipeline
  ::Ci::CancelPipelineService.new(pipeline: pl, current_user: w.user).force_execute
  ::Ai::DuoWorkflows::UpdateWorkflowStatusService.new(workflow: w.reload, status_event: 'stop', current_user: w.user).execute
  puts "stopped: session #{w.reload.status_name}, workload pipeline #{pl.reload.status}"
end

  4. A plain REST caller, which is what moving this into the route adds. The same call against both ports:

    for host in gdk.test:3000 127.0.0.1:3021; do echo "== $host"; curl -s -X POST "http://$host/api/v4/ai/duo_workflows/workflows" \
  -H "PRIVATE-TOKEN: $API" -H "Content-Type: application/json" \
  -d '{"project_id":"gitlab-duo/test","workflow_definition":"fix_pipeline/v1","goal":"http://gdk.test:3000/gitlab-duo/test/-/pipelines/1625","start_workflow":true,"environment":"ambient"}' --max-time 300 \
  | jq -c '{id, status, workload: .workload.id, message} // .'; done

    What I got:

    == gdk.test:3000
{"id":1031,"status":"created","workload":550,"message":null}
== 127.0.0.1:3021
{"id":1032,"status":"created","workload":551,"message":null}

    And from inspect_session.rb, master (1031) then this branch (1032):

    session 1031 status=failed definition=fix_pipeline/v1 source_type=
linked merge request: !172  source pipeline: 1625
DUO_WORKFLOW_SOURCE_BRANCH=main
workload checks out 0e2d7118 (MR !172 head 47f830cb, main 0e2d7118)
no merge_request or pipeline context
    session 1032 status=running definition=fix_pipeline/v1 source_type=
linked merge request: !172  source pipeline: 1625
DUO_WORKFLOW_SOURCE_BRANCH=mcp-fix-pipeline-631689
workload checks out 47f830cb (MR !172 head 47f830cb, main 0e2d7118)
context merge_request: {"url":"http://gdk.test:3000/gitlab-duo/test/-/merge_requests/172","author_id":"1"}
context pipeline: {"source_branch":"mcp-fix-pipeline-631689","source":"merge_request_event"}

  5. A caller that sends the branch and context itself, as the UI does, gets exactly what it sent:

    curl -s -X POST "http://127.0.0.1:3021/api/v4/ai/duo_workflows/workflows" -H "PRIVATE-TOKEN: $API" -H "Content-Type: application/json" \
  -d '{"project_id":"gitlab-duo/test","workflow_definition":"fix_pipeline/v1","goal":"http://gdk.test:3000/gitlab-duo/test/-/pipelines/1625","start_workflow":true,"environment":"ambient","source_branch":"mcp-fix-pipeline-631689","additional_context":[{"Category":"merge_request","Content":"{\"url\":\"sent-by-the-caller\"}"}]}' --max-time 300 | jq -c '{id, status} // .'

    What I got: {"id":1033,"status":"created"}, and from inspect_session.rb:

    DUO_WORKFLOW_SOURCE_BRANCH=mcp-fix-pipeline-631689
context merge_request: {"url":"sent-by-the-caller"}

  6. Another project's pipeline. Over MCP it is refused before anything starts, with the project's session count at 919 before and after:

    curl -s -X POST http://127.0.0.1:3021/api/v4/mcp -H "Authorization: Bearer $MCP" -H "Content-Type: application/json" -H "X-Gitlab-Enabled-Mcp-Server-Toolsets: duo_agent_platform" \
  -d '{"jsonrpc":"2.0","method":"tools/call","id":1,"params":{"name":"start_duo_session","arguments":{"project_id":"gitlab-duo/test","flow":"fix_pipeline/v1","goal":"http://gdk.test:3000/free-self-managed-group/project/-/pipelines/1618"}}}' --max-time 300 | jq -c '.result // .'
    {"content":[{"type":"text","text":"400 Bad request - fix_pipeline/v1 fixes a pipeline, so the goal must be the URL of a pipeline in gitlab-duo/test"}],"structuredContent":{"error":{"message":"400 Bad request - fix_pipeline/v1 fixes a pipeline, so the goal must be the URL of a pipeline in gitlab-duo/test"}},"isError":true}

    Over REST it starts as it did before (session 1034), on main with no context:

    curl -s -X POST "http://127.0.0.1:3021/api/v4/ai/duo_workflows/workflows" -H "PRIVATE-TOKEN: $API" -H "Content-Type: application/json" \
  -d '{"project_id":"gitlab-duo/test","workflow_definition":"fix_pipeline/v1","goal":"http://gdk.test:3000/free-self-managed-group/project/-/pipelines/1618","start_workflow":true,"environment":"ambient"}' --max-time 300 | jq -c '{id, status} // .'
    DUO_WORKFLOW_SOURCE_BRANCH=main
no merge_request or pipeline context

    On master (port 3000), both calls start a session on main with no context (sessions 1036 and 1037).

  7. The epic's journey, step 1.10, as a client would run it. Claude Code gets only the merge request link and has to find the failed pipeline itself, against the branch's MCP server:

    {"mcpServers":{"gdk-duo":{"type":"http","url":"http://127.0.0.1:3021/api/v4/mcp",
  "headers":{"Authorization":"Bearer ${GDK_MCP_TOKEN}","X-Gitlab-Enabled-Mcp-Server-Toolsets":"duo_agent_platform"}}}}
    GDK_MCP_TOKEN="$MCP" claude -p "The pipeline on http://gdk.test:3000/gitlab-duo/test/-/merge_requests/172 failed. Start Duo's fix pipeline flow for it, then tell me the session URL. Do not do anything else." --mcp-config mcp-gdk.json --strict-mcp-config --allowedTools "mcp__gdk-duo__start_duo_session,mcp__gdk-duo__get_merge_request,mcp__gdk-duo__list_pipelines,mcp__gdk-duo__get_pipeline" --output-format stream-json --verbose

    The calls it made, get_merge_request and then start_duo_session:

    {"url":"http://gdk.test:3000/gitlab-duo/test/-/merge_requests/172","include":["pipelines"],"pipelines_first":5}
{"flow":"fix_pipeline/v1","goal":"http://gdk.test:3000/gitlab-duo/test/-/pipelines/1625","project_id":"gitlab-duo/test"}

    Session 1035 finished a few minutes later, and get_duo_session returned a diff note on .../merge_requests/172#note_2329: a suggestion on scripts/greet.sh by the flow's service account. To see that it used the merge request rather than finding it on its own:

    p Ai::DuoWorkflows::CheckpointBlob.where(workflow_id: 1035).distinct.pluck(:channel).include?('branch:to:fetch_mr_diffs#deterministic_step')
# => true

    The router skips fetch_mr_diffs when there is no merge request URL, so a session started on master does not have that step.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Fred de Gier

Merge request reports