Clarify reply delivery in Developer mention conversations

Thomas Schmidtrequested to merge
duo/fix/2804-mention-conversation-context into master

What does this MR do and why?

When mentioning Duo Developer, it sometimes posts its findings to the discussion itself, and the platform then delivers the agent's final answer a second time, producing two near-duplicate comments from the service account. This MR is a prompt- and context-level attempt to reduce that behavior. The real check is monitoring the duplicate-pair rate after rollout (follow-up in the linked issue).

Two changes:

  1. Goal template (Developer mention only). Context and delivery guidance now come before the unchanged XML transcript, and the goal ends with the conversation. The third-person introduction ("@user is talking to you in a conversation on this issue") becomes metadata (Requesting user: @user).
  2. Note-history cutoff (all note-triggered flows using Ai::Notes::ConversationContextBuilder). History is now bounded at the triggering note by (created_at, id) before the note cap applies. Main reason: the transcript always ends with the message being answered, and later comments can no longer displace the trigger from its own prompt when a thread exceeds the cap. As a side effect, this also closes the window in which a comment posted while the trigger is being processed could slip into the prompt. Complete and trimmed history notices describe the boundary; conversation budgeting is unchanged.

The gateway's "persist your results to GitLab" wording was clarified separately in ai-assist!6845.

References

How to set up and validate locally

End-to-end (GDK)

  1. Check out this branch in your GDK with the local AI gateway running and the foundational Developer flow available.

  2. In a test project (for example gitlab-duo/test), post a comment on an issue mentioning the Duo Developer service account

    Useful task shapes:

    • A read-only question ("Summarize this repository's structure").
    • A task that asks for posted output, e.g. "Investigate X, post your findings as a comment, and come back with a TLDR" — expect the findings comment and a distinct short delivered reply.
    • A small coding task ending in a merge request.
    • A mention on a merge request instead of an issue.

  3. Open the session from the "started session" note and let it complete.

  4. In the discussion, verify:

    • The task is done well; for coding tasks the MR is opened and assigned to you.
    • No duplicate pair: the platform-delivered reply (it carries the session link) must not repeat a comment the agent posted itself. A posted artifact plus a distinct short confirmation is the desired pattern; the same content twice is the failure mode.
    • The agent still uses tools freely for the work itself (branches, commits, MRs, and posting comments when asked).

While a session runs, you can also post a non-mention follow-up comment in the same discussion: the transcript rendered for the agent should still end at the triggering message.

Local validation results of different sanity tests

Local validation results (GDK, this branch)

Three wording iterations of the delivery rule were tested on gitlab-duo/test, scored against "duplicate pair = failure" (self-posting distinct content is desired flexibility). Goals verified in each session's stored goal; similarity measured between self-posted and delivered notes.

Round 1 (strict never-post wording), issues 15–17 / MR !218 (closed): unambiguous asks clean (3015, 3020); posting-shaped asks produced the desired split while violating the letter of the rule (3018 similarity 0.27, 3019 similarity 0.12); one verbatim duplicate pair on a neutral "assess this MR" ask (3017, similarity 0.97); coding task clean (3016, MR !218 (closed) opened and assigned).

Round 2 (duplication discipline): posting-shaped asks reliably produced the desired split (3052, 3055); residual failure mode was short answers — nothing to summarize, so "keep it distinct" cannot apply (3056: short assessment self-posted, then delivered reworded).

Round 3 (final wording: delivered response by default + duplication discipline), issues 19–22 / MR !218 (closed):

Scenario Session Result
"Research, post findings, come back with a TLDR" 3052-shape rerun: 3066 ✅ findings posted + distinct TLDR delivered (0.03)
"Review this MR for correctness" 3068 ✅ review posted + distinct confirmation delivered (0.12)
"Brief assessment: what does it change" (MR) 3069 ✅ single delivered reply
"What does this MR change? Brief assessment" — the round-2 failure wording 3070 ✅ single delivered reply
Neutral two-sentence summary (issue) 3067 ✅ single delivered reply
Long neutral investigation (CI failure analysis, multi-file + git history) 3071 ✅ no duplicate pair — full report posted, distinct summary delivered (0.15). Note: the agent self-posted the long-form report without being asked; acceptable under the decided semantics (distinct content), worth watching in production.

The note-history cutoff was verified live: a transcript built from a triggering note ends at that note and excludes the later platform-delivered reply, with the full-thread notice stating the boundary.

Bottom line: with the final wording, zero duplicate pairs in six round-3 sessions, the previously failing shape included. Ablations (glab skill removed) showed the skill supplies the posting recipe at decision time but is not itself the problem to fix — the neutral skill stays untouched. Small n per shape, one local setup, one model: a smoke signal, not proof. Production monitoring of the duplicate-pair rate after rollout is the real measure.

Note: the local GDK runs a local AI gateway that already includes the gateway-side wording change (ai-assist!6845), so local sessions exercise both changes together.

Database Review

Postgres AI Link (I hope I correctly mapped this note as an example).

Query on my local gdk:

SELECT "notes".* FROM "notes"
WHERE "notes"."noteable_type" = 'Issue'
  AND "notes"."noteable_id" = 3585
  AND "notes"."discussion_id" = 'd35fd68992b9f5efd7c9cecf49f4e77bef226883'
  AND ((notes.created_at, notes.id) <= ('2026-09-11 13:06:34.730562', 10656))  -- this line is added newly
  AND "notes"."system" = FALSE
ORDER BY "notes"."created_at" DESC, "notes"."id" DESC
LIMIT 16;

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

  • Local code review: no blockers found for opening this draft.
  • Targeted specs and surrounding consumer specs pass.
  • RuboCop passes on all changed files.
  • Changelog and EE commit trailers included.
  • Local GDK smoke validation completed (three wording iterations; final wording: zero duplicate pairs in six sessions).
  • Full SQL, representative plans, and database review completed.
  • CI and rollout readiness reviewed before removing draft status.
Edited by Thomas Schmidt

Merge request reports