Add image attachments to Duo Agentic Chat

Igor Drozdovrequested to merge
duo-agentic-chat-image-attachments into master

What does this MR do and why?

Users can now attach images to a Duo Agentic Chat message via the attach action in the + menu, by pasting, or by dragging files onto the prompt area.

Images travel to the flow service as additional_context entries under an attachments category, one entry per file, whose content is a JSON document holding the base64 payload. This piggybacks on the existing context channel because AdditionalContext is four string fields and StartWorkflowRequest has no bytes field, so there is no binary path to use instead. This mirrors how orbit_context and form_context already flow through the WebSocket start request.

This is mostly frontend, but it does need a small Rails change beyond the flag definition and its push_frontend_feature_flag call. Sending is indeed free-form — the WebSocket path forwards additional_context straight through, with no allowlist in the way — but reading a transcript back is not, and that is why the attachments category is registered in Ai::AdditionalContext::DUO_CHAT_CONTEXT_CATEGORIES. The detail is in the design notes below.

The feature is behind the dap_web_chat_file_attachments experiment flag (default_enabled: false, milestone 19.4). Every path a file can take into the composer is gated: the "Attach image" menu item, the hidden file input, the file picker, paste, and drag and drop. The + dropdown itself renders only when either this flag or the pre-existing dap_web_search flag is on, so with both off the interface is identical to what it was before this work. No AI Gateway flag is pushed, because the gateway MR has no flag check of its own; the gating is entirely client side, and nothing reaches the gateway unless the composer produces attachments.

Design notes worth reviewer attention

Limits are duplicated on purpose. The size, count, and MIME-type limits in attachment_utils.js deliberately mirror duo_workflow_service/entities/attachments.py (5 files, 2 MiB each, 2,500,000 bytes total, PNG/JPEG/WebP). The service re-validates everything, and an attachment that falls outside those bounds fails the whole turn rather than just that one file, so the user's message is refused outright instead of being sent without the offending attachment. A looser limit here would therefore cost the user everything they had typed, where the client's own check would have shown a per-file warning and let the rest of the message through. A spec asserts the constants explicitly so a future divergence fails loudly.

The attachment reference token comes from the service, not from the client. An earlier approach pinned extras.contextItems onto the pending user message on the client, and it never reliably worked: when the turn comes back from the flow service as a ui_chat_log entry, transformChatMessages rebuilds extras from the echoed additional_context, and the Vuex ADD_MESSAGE mutation merges shallowly, so the incoming extras replaced the pinned one wholesale. The token survived only on a turn that carried no other context, because then the echoed context was empty and extras was never rebuilt. Instead, the flow service echoes a payload-free reference envelope naming each attachment on the turn's UiChatLog, and the client builds nothing — the token arrives with every other included reference. This also means the reference survives a page reload, which a client-side token never could, because the transcript is rebuilt from the service checkpoint. The resulting behaviour matches the existing "Current page" context reference, which likewise only appears once the turn has round-tripped.

The payload-free envelope still has to satisfy the Rails GraphQL contract. The reference envelope the flow service echoes back failed that contract twice. Reading a transcript goes through Types::Ai::AdditionalContextType, whose category is an AiAdditionalContextCategory enum generated from Ai::AdditionalContext::DUO_CHAT_CONTEXT_CATEGORIES, so an unregistered attachments category raised an unresolved value error that nothing rescues, failing the whole query rather than the one item; hence the registration. Its content is non-nullable and the envelope deliberately carries no payload, so the flow service now sends an empty string rather than leaving it unset. Both failed only on a later read of the checkpoint, not on the turn itself, because the send path is a websocket that never touches Rails. Registering the category has a consequence worth noting: the same constant drives UserAvailableFeaturesResolver, which checks a duo_include_context_<category> feature flag for every category unless it is listed as already enabled, and maps each to an include_<category>_context unit primitive. attachments is added to the already-enabled list, because the feature has its own experiment flag and a flag with no definition file would raise, and unlike every other category it registers no unit primitive in the cloud connector, because attachments carry user-uploaded image bytes rather than a permission-gated context source.

Images cannot be shown as message content. duo_chat_message.vue renders user messages inline through its markdown renderer and only delegates to custom messageRenderers in the v-else branch for other roles, so a custom user-message renderer would never be reached. That renderer also escapes every <img>. The attachments category is deliberately not one of duo-ui's openable categories (file, local_git, dependency, terminal), which keeps the token inert instead of opening a details modal we have no content to fill.

An image on its own is a valid message. goal has no presence validation (only GOAL_MAX_LENGTH), and the service appends image blocks to history independently of the prompt text, so submit is enabled with attachments and no text.

The attachments ride on the draft prompt. This MR is rebased onto the PromptComposer refactor, !253612 (merged), so all of the file handling lives in one component, prompt_composer/file_attachments.vue: the picker, paste, drop, validation, the rejection notices and the previews. The attachments themselves live on the composer's draft, added and removed through UserPromptBuilder's withAttachment and withoutAttachment, so a single object carries the whole prompt; the component reads them as a prop and asks for changes rather than holding its own copy. send-chat-prompt therefore keeps its single argument, since the attachments travel on the UserPrompt. isDraftSubmittable stays the single place deciding whether a draft is worth sending, so the submit button and the Enter key cannot disagree, and it now also accepts a draft with attachments and no text. The form binds the paste and drag events, because it owns the element they land on, and forwards them to the component.

Defects fixed in the unshipped code

  • Submitting with attachments but no text while a turn was already running queued an empty prompt, which would later be sent to the service as an empty goal. This was reachable with the Enter key. The composer now declines the submission and holds the attachments for the next sendable turn, because the prompt queue carries text only and cannot carry attachments.
  • The "Web search" row in the + menu rendered bold while "Attach image" did not. The bold comes from GlToggle's .gl-toggle-label class, which styles the label as a form label (bold, text-strong). That is correct in a settings form and wrong in a menu, where the label is just the item's name. It is now overridden on the label slot so both rows match.

References

Screenshots or screen recordings

Before After

How to set up and validate locally

  1. Enable the feature flag in the rails console:

    Feature.enable(:dap_web_chat_file_attachments)

  2. Run an AI Gateway that includes gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!6731 (merged), and point your GDK at it.

  3. Open Duo Agentic Chat in the web UI.

  4. Attach an image three ways, and confirm a preview chip with thumbnail, filename, and size appears for each:

    • click + and choose Attach image
    • copy an image to the clipboard and paste into the prompt box
    • drag an image file onto the prompt area and confirm the drop overlay appears

  5. Remove one attachment with its x button.

  6. Send with no prompt text and confirm the message sends, and that the user turn shows a paperclip token per file.

  7. Ask a question about the image contents and confirm the model can see it.

  8. Check the rejection paths:

    • a non-image file (for example a PDF) is rejected with a dismissible warning, while other files in the same batch still attach
    • a file over 2 MiB is rejected
    • a sixth file is rejected with the count message

  9. Disable the flag and confirm the + menu shows no Attach image item, and that pasting or dragging an image does nothing.

Automated coverage

yarn jest ee/spec/frontend/ai/duo_agentic_chat

84 suites / 2013 tests pass, under both Vue 2 and Vue 3. Most of that total predates this MR.

Attachment-specific coverage is 93 tests. 66 of them are in three dedicated suites: attachment_utils_spec.js (32), file_attachments_spec.js (26) and attachment_previews_spec.js (8). The other 27 are in the attachments describes inside prompt_composer_spec.js (13), prompt_input_actions_spec.js (6), duo_agentic_chat_state_manager_spec.js (5) and duo_agentic_chat_state_manager_light_spec.js (3). Between them they cover the limits, the snake_case wire keys, per-file, total and count rejection with partial-batch acceptance, paste versus plain-text paste, nested dragenter/dragleave counting, attachment-only submit, refusing to queue a draft holding images, clearing after send, and the flag-off state for every entry point.

Known gaps

  • Base64 inflates the payload by roughly 33%, so a full 2.5 MB of attachments becomes ~3.3 MB on the wire. I have not verified that Workhorse or the WebSocket start path accepts that; if there is a lower ceiling it would fail opaquely rather than through the friendly per-file errors. Worth a reviewer's eye.
  • An image is visible to the model only until the first interrupt of the turn it was sent on, not just on later turns. Stripping runs on every checkpoint write and a resume rebuilds history from the checkpoint, so a mid-turn tool approval is enough to lose it. The filename token survives, so the model can say which file it can no longer see rather than guessing. Durable fix tracked in gitlab-org/modelops/applied-ml/code-suggestions/ai-assist#2826.
  • The reference token depends on the gateway MR being deployed. Until then, no attachment reference renders on the sent message at all.

Screenshots

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Igor Drozdov

Merge request reports