## Summary Build a system that semi-automatically refine work items to become ready to picked up by Agents to work on. This is a foundational set of primitives that are required _before_ making the future decision to invest in a long-lived spec object layer, which is being [explored separately](https://gitlab.com/gitlab-org/plan-stage/software-factory-exploration/-/work_items/4). Throughout this epic, **"spec" refers to the work item's description** — the field capturing the Why (motivation) and What (requirements) of the work. The How is captured in the AI-generated Work Plan. ## Problem In every software engineering teams, there is always a significant backlog of issues that are easy, low hanging fruits that are trivial, but takes long times to be addressed by teams due to other more pressing priorities. ## Solution We want to address this issue by essentially proposing to shrink any dev team backlog by roughly 25% by having Agents able to autonomously refine and fix issues. ## User journey - Prototype: https://plan-sdd-bdc021.gitlab.io/ - Latest walkthrough: https://gitlab.com/groups/gitlab-org/-/work_items/22703#note_3754968901 - PoC for fully autonomous execution: https://www.youtube.com/watch?v=yQ_ystCPjFk ## Workstreams - https://gitlab.com/groups/gitlab-org/-/work_items/22774+ - https://gitlab.com/groups/gitlab-org/-/work_items/22703+ - https://gitlab.com/groups/gitlab-org/-/work_items/22702+ - https://gitlab.com/groups/gitlab-org/-/work_items/22772+ - https://gitlab.com/groups/gitlab-org/-/work_items/21552+ - https://gitlab.com/groups/gitlab-org/-/work_items/23050+ - https://gitlab.com/groups/gitlab-org/-/work_items/22773+ ## Requirements ### **Backlog refinement loop (Out of scope)** To have truly a full automation for the backlog refinement, we'd want a flow that can estimate effort (weight) issues to _find_ the low hanging fruits in the first place. There is some level of complexity because weighting, even for humans, is _hard_. To keep in mind here: 1. Weighting is **not** refinement. We wouldn't send agents to read the entire repo and come up wit ha plan because that is the second loop, so we'd want a low token cost way to find these. This is **_out of scope_** for the release in October for Transcend. ### Work item refinement loop Today we have to make the work plan interactively with Duo Chat. That means that there is a direct linear correlation of users to work plan (1 user can make a single plan at a time). To have any chance to parallelize the refinement work, we need them to be async (a single user can kick off a few) and then get notified if their input is required on any of them. To realize this, there are a few requirements: 1. Agents **must** make their own suggestions. Meaning when a question arise, we don't want to stop the whole flow. Agents list the questions and then explicitly give 2-3 options per questions, then confirm what **_their_** recommendation _is._ 1. This is so that we can support YOLO mode (I trust the agent suggestions, I don't have to read) **and** more gated approach of getting a work plan and then iterating on it before passing of to Duo developer. 2. Notifications are **mandatory**. When an agent requires something from you, user must be pinged. The form of this communcaition is up to confirmation, but there is a strong case to start with Agents _commenting_ on their issue and pinging the author when an action is required. 3. The Generated work plan contain as much research upfront as possible to remove the cost of higher models on the implementation side _and_ so that humans can review the plan more accurately before handoff. 4. The Work Plan takes into account the possibility of parallel work. If certain tasks should be done by different sub-Agent, it brings it up in its plan. ### Benchmarking We need to make sure that we have good benchmarking in place that will allow us to validate the assumptions that work plans save money and produce better output. Then when we have the work item refinement loop, we can trigger this specific loop to test how well it integrate with existing SWE benchmarks. ## Constraints * No hierarchies in this iteration. Work Items with children cannot have Work Plans. * No live collaborative chat. Only through comments. ## Timeline, sequencing, and goals <table> <tr> <th></th> <th>Focus</th> <th>Outcome</th> </tr> <tr> <td>August 31</td> <td> * Instrument and collect necessary telemetry to track loop lifecycle and capture credit consumption alongside events. * Loop polish. Multiple entry points: intent to new work item, refine existing work item. * Confidence score feeds context back into the intent and plan until it reaches the "good to go" threshold. * "Notes to Threads" transition is validated by the wider user-base. * Benchmark/eval Phase 1 * Evangelize internally, launch pilot, collect and synthesize usage data. * Clickable prototype for Design Partner validation. </td> <td> "_As a solo engineer_, _I can confidently resolve less complex backlog items one at a time using the loop with minimal human intervention and trust the code generated will resolve the backlog item_" * **Customer Zero:** Pilot -- One team from each department (Core DevOps, AI, UX Papercuts, ...) resolves 5+ P3/P4 bugs from their backlog with the intent-to-code loop and provides experience feedback (Problem/solution fit). * **Design Partners:** 5 identified, committed, and clickable prototype validated with each. * **Data Validation**: Phase 1 - [Benchmark/eval](https://gitlab.com/groups/gitlab-org/-/work_items/22774) (cost, correctness, ...), real-world engineering metrics derived from Customer Zero pilot. </td> </tr> <tr> <td>October 1</td> <td> * Decision log * Polish mini-loops (Refine, Plan, Implement) * Streamlined interactions with Duo Chat (refinement questions, user feedback on progress and current task, reduce approval overhead, ...) * AI Triggers to incorporate mini-loop steps (or the entire loop) into existing workflows (e.g. Board transition) * New threaded convo experience is live behind a FF * .... </td> <td> "_As an engineer on a team_, _I can efficiently delegate many backlog items to the loop to work autonomously, understand when I need to take an action to keep forward progress, understand what decisions have been made, and collaborate with my team asynchronously to bring my unique expertise to the highest priority task at hand._" * **Customer Zero:** Highest-priority adoption blockers resolved. * **Design Partners:** Begin \~3 month closed beta. Solution and user experience meet the minimum quality bar for active experimentation with 5 design partners. * **Data Validation:** Phase 2 - Benchmark/eval and real-world engineering metrics prove customer value and business impact. </td> </tr> <tr> <td>November 30</td> <td> * .... </td> <td> "_As a product manager on a team_, _I can collaborate with Duo to capture and refine more complex, multi-step requirements into actionable workstreams my engineers can pick up and execute on. As decisions are made, scope changes, and implementation progresses, everyone on my team effortlessly stays aligned and has shared, transparent context._" * **Customer Zero:** Consistent week-over-week growth in Weekly Active Users (WAU) and count of work items resolved with the intent-to-code loop. * **Design Partners**: First wave of highest priority adoption blockers resolved. \~75% "solution-market" fit. * **Data Validation:** Phase 3 - Benchmark/eval and real-world engineering metrics improve from Phase 2 baseline. </td> </tr> <tr> <td>December 30</td> <td> * .... </td> <td> "_As a/n \[insert role\] on a delivery team, agents drive the majority of the work, only requiring human intervention when certain guardrails we put in place are triggered. Instead of a backlog of work to be executed, we build and ship software by writing, and evolving, long-lived specs. We still review code, but have high confidence and trust in the output of the agents. Collaboration on the specs and higher-level system design is where we spend the bulk of our time._" * **Customer Zero:** Consistent week-over-week growth in Weekly Active Users (WAU) and count of work items resolved with the intent-to-code loop. * **Design Partners:** Lovable sentiment. Could not imagine a GitLab that did not support this capability and solution. * **Data Validation:** Phase 4 - Benchmark/eval and real-world engineering metrics improve from Phase 3 baseline. * **Go To Market:** Initial GTM plan and positioning alignment with PMM and the Field. </td> </tr> <tr> <td>January 14</td> <td></td> <td> * **Minimum Lovable Product:** Quality bar achieved. * **Go To Market:** Locked in and ready to go. * </td> </tr> </table> ## Experience spine Scope split. Steps 1 to 3, 7, and 8 are the single-player path and fit this current backlog epic, whose journey is solo and whose constraints exclude hierarchies. Steps 4 to 6 are the team and multiplayer experience (threads, child workstreams, suggested owners) and need collaboration and hierarchy in scope, so they likely belong in a multiplayer sub-epic. 1. Start an intent from chat or backlog. Workstream: entry points. * Critique question: is starting frictionless for a new user. * Expectations: start in chat for a net-new idea, or navigate to an existing work item from the Duo chat session. Nice to have: reach the existing item from a saved view or board in under two clicks. 2. Duo asks context-based questions. Workstream: intent Q&A. * Critique question: are the questions smart and context-aware, or generic. * Expectations: this could be the 1st magic moment. Ask relevant questions about the idea, not "do you want a work item created," and don't ask for approval five times before the work item exists. 3. Duo writes the work item. Workstream: planning-agent quality. * Critique question: is the spec actually good? * Expectations: benchmarking is needed to back this; assume humans review first until trust is earned; were the right things captured, is it well written and clear, does it capture description and goal, how long did the creation take (seconds or minutes). 4. Tag a teammate, assume async back-and-forth. Workstream: threads. * This is the Oct multiplayer mode and a differentiator we need to keep. * Critique question: does collaboration feel natural and can survive the move off comments. * Expectations: threads should be easy to find, how do I add comments to them easily, clear signals for me to check them. \[_yes, first design unblock needed\]_ 5. Duo breaks into child workstreams, humans can edit. Workstream: effort breakdown. * Critique question: is the breakdown helpful and sensible (users may not yet know how to divide the work). Trust can be earned quickly here. 6. ~~Duo suggests owners based on who weighed in, humans assign. Workstream: suggested owners. Stretch, not needed for Oct. A second multiplayer moment, distinct from step 4.~~ * ~~Critique question: suggestions must be credible.~~ 7. Plans are built, then handed off for readiness assessment. Workstream: workplan. * Critique question: is the plan complete and trustworthy (contingent on step 3). * Note: readiness scoring is now tracked separately in [WS-6: Work Plan Readiness Score](https://gitlab.com/groups/gitlab-org/-/work_items/23050). 8. Human gives the go, future agent proceeds. Workstream: handoff. * Critique question: does handing off to code feel safe.