## Summary Introduce a **score for the Agent Plan (WorkPlan)** that communicates how ready a plan is for agent execution. This score acts as a lightweight quality gate — giving humans and automation a signal about whether a WorkPlan is sufficiently refined before kicking off downstream agent work (e.g., Duo Developer handoff). This issue covers the design, backend, and frontend work needed to compute, display, and act on the score within the WorkPlan widget. **TBD: This issue should also include an approval plan or keep it in mind.** ## Parent Epic https://gitlab.com/groups/gitlab-org/-/work_items/21218 ## Related - Workstream 0 — Agent Plan: https://gitlab.com/groups/gitlab-org/-/work_items/21511 - Workstream 1 — MR spec enforcement: https://gitlab.com/groups/gitlab-org/-/work_items/21514 - Interactive Builder: https://gitlab.com/groups/gitlab-org/-/work_items/21653 - UX Research — Enforced Workflows: https://gitlab.com/gitlab-org/ux-research/-/work_items/3256 ## Problem Today, anyone with a Duo seat can trigger an agent handoff (e.g., issue-to-MR) regardless of how well-defined the underlying WorkPlan is. This creates two risks: 1. **Wasted compute/tokens** — poorly scoped plans lead to low-quality agent output, burning tokens with little value. 2. **No quality signal** — there is no mechanism to tell a user (or an automated policy) whether a WorkPlan is "good enough" to hand off, leading to inconsistent outcomes. Customers have expressed a need for approval checkpoints across the SDLC (see tollgate methodology pattern). However, heavy-handed enforced workflows slow down cycle time. A score-based approach provides a lightweight, automatable alternative that balances quality control with speed. ## Proposed Solution ### Naming The working name for this metric is **"Readiness Score"** (alternatives discussed: Confidence Score, Alignment Score, Implementation Score). Final naming TBD with UX — the score should semantically convey "how ready is this plan for agent execution" rather than "how well does the output align with the plan" (which is a downstream concern). ### Core Concept A numeric score (e.g., 0–100) displayed on the WorkPlan widget that reflects how well-defined and complete the plan is for agent consumption. The score is computed by evaluating the plan content against a set of criteria. ### Behaviour | Score Range | UX Treatment | Handoff Behaviour (v1) | |---|---|---| | High (e.g., ≥ 70/100) | Green indicator — plan is ready | User can kick off agent execution without friction | | Medium (e.g., 40–69) | Yellow indicator — plan needs refinement | Soft warning dialog: _"The readiness score is only X/100. Would you like to refine the plan further before kicking off?"_ | | Low (e.g., < 40) | Red indicator — plan is incomplete | Stronger warning; suggest entering plan refinement mode (e.g., Interactive Builder or Q&A flow) | **v1 is intentionally lightweight:** the score is advisory, not blocking. All users can still proceed regardless of score. This avoids introducing hard approval gates prematurely while still protecting teams from burning tokens on under-specified plans. ### Scoring Criteria (to be refined) Potential signals for computing the score: - **Completeness** — Are key sections of the plan populated (requirements, acceptance criteria, constraints, context)? - **Clarity** — Is the language unambiguous and structured enough for agent consumption? - **Scope** — Is the plan appropriately scoped (not too broad, not too narrow)? - **Context richness** — Does the plan reference relevant artifacts (linked issues, design docs, decision log entries)? Scoring criteria could eventually be **configurable by project owners**, allowing teams to define what "ready" means for their context. ### Future Iterations (out of scope for v1) - **Hard approval gates**: Only users with specific roles can approve a handoff when the score is below a threshold (connects to customer requests for enforced workflows). - **Score-based automation policies**: Project-level settings like _"Auto-approve handoff if score ≥ 80; require DRI approval if < 80."_ - **Score history / trend**: Track how the score evolves as the plan is refined. - **Downstream scoring**: Separate score for how well the MR output aligns with the plan (distinct from this readiness score). - **Integration with tollgate/stage-gate workflows**: The score mechanism can serve as one checkpoint in a broader approval flow across the SDLC (requirements → development → QA → deployment). ## Architecture Considerations The architecture chosen for the WorkPlan (see discussion in Workstream 0) has implications for scoring: - If the WorkPlan uses a **git-backed storage model** (e.g., markdown files with versioning akin to Rapid Diffs), scoring can leverage diffs, version history, and commenting — making it easier to show _why_ a score changed and to build review/approval flows on top. - If the WorkPlan is stored as a **BLOB in PostgreSQL**, scoring is still feasible but versioning, diffing, and audit trails become harder to implement later. This issue should be mindful of the storage decision being made in Workstream 0 and ensure the scoring approach is compatible with the chosen architecture. ## UX Requirements - [ ] Display the score prominently on the WorkPlan widget (exact placement TBD with design) - [ ] Provide a clear visual indicator (color-coded or icon-based) for score ranges - [ ] Maybe: Show a soft-gate dialog when a user attempts to kick off agent execution with a low score - [ ] Maybe: Provide a way for users to understand _why_ the score is what it is (breakdown or tooltip) - [ ] Ensure the score updates dynamically as the plan is edited or regenerated ## Open Questions 1. **Scoring model**: Should the score be computed by an LLM call, a heuristic/rule-based system, or a hybrid? What are the cost/latency tradeoffs? 2. **Configurability**: How much control should project owners have over scoring criteria in v1 vs. later iterations? 3. **Score persistence**: Should we store score history, or only the current score? 4. **Threshold defaults**: What are sensible default thresholds for the soft-gate dialog? 5. **Naming**: Final decision on "Readiness Score" vs. "Confidence Score" vs. other options. 6. **Scope boundary**: Where does this score end and downstream alignment/enforcement scoring begin?