## Original Ask TL;DR: - https://claude.ai/artifact/8TM5koJvxUgut1omnwehdw?sk=dGSp8je-tt6TnAcULVhc1Q is a great start. It's most valuable to most of this data in time-series week-over-week charts to understand adoption/churn trends over time (by team, by type, ....) - Unique users as well as total event counts and avg. workplans created per user week-over-week are good adoption signals. - Distribution of async vs. sync flows (maybe `User Agent`?) - Utilizing different capabilities in the flow via weekly time series -- decisions recorded, plans generated, implementation initiated, readiness score reviewed (maybe tooltip or whatever action shows the readiness score context), ... - Average number of workplan revisions - It's fine to scope this to customer zero-related namespaces only (gitlab-org/gitlab-com) - Failure rates over time (unknown errors) ### Things that would be interesting - Funnel analysis: Open Work Item > Click `Plan` > Plan viewed > Click `Implement` > MR Merged (% of users that progress from one step to the next as well as the average duration between steps) - Comparison analysis -- Difference between MRs created from workplans vs. non-workplan MRs: - merge rate (merged / merged + closed) - average changes requested by reviewers - wall clock -- time from MR created to MR merged - avg. review time -- start review to submit review - pipeline failure rate (do workplan generated MRs fail less than not workplan generated) - Average commits per MR - Anything else that is commonly used as some sort of proxy metric to understand/measure engineering productivity ## Agentic plan tracking: build plan Living copy: [build plan doc](https://claude.ai/code/artifact/82d04e62-56c8-4f2c-be79-5dd96158a869). ### Goal and scope We want to measure how much the workplan flow (`workplan/v1`) is used, and what happens to the Developer flow runs (`developer/v1`) that implement a plan. The data is for GitLab internal use only. The plan should answer these questions: weekly users and workplans per user; the top-N users who create workplans and use them to implement MRs, with the MR outcome (merged, closed, open); plans generated; implementations started; decisions recorded; readiness score viewed; plan revisions; failure rates; whether a run read the plan; workplan MRs versus other MRs; the client split; and cost per run. This doc lists only what we will build (W1-W5) and nice-to-have follow-ups (F1-F11). The reasons other ideas were left out are in the [decision log](https://claude.ai/code/artifact/c6b6adaf-9176-4a4a-9a6c-609e269b500c). ### Metrics and how each is built Most metrics work from data already in Snowflake. The last column names the work item that fills each gap. | Metric | Source today | Needs | | --- | --- | --- | | Weekly users, workplans per user | `GITLAB_DOTCOM_DUO_WORKFLOWS_WORKFLOWS` rows by `DUO_WORKFLOW_DEFINITION` and `USER_ID` | W1 for team and type from issue labels | | Top-N creators and implementers with MR outcome | Workflow rows plus `GITLAB_DOTCOM_DUO_WORKFLOWS_WORKFLOW_MERGE_REQUESTS` (link type 1) and the MR tables | W1, to tie the workplan run and the developer run to the same work item | | Plans generated | `workplan/v1` run counts | W2 for plans per work item | | Implementation started | `agent_platform_session_created` (source shared with the header button) | W5 for new runs, W1 (`is_goal_workplan_prefixed`) for older runs | | Decisions recorded | None in PROD | W2. The agent vs human split needs `work_item_decisions.workflow_id`, which the application does not populate today | | Readiness score viewed | None | W4 | | Plan revisions | `work_item_agent_plan_update` events split by `source` | W3 for the work item id | | Failure rates | Workflow status, `agent_platform_session_dropped` for abandoned runs, `ai_context` `flow_name` and `item_version` for per-version splits | Nothing | | Run read the plan | `work_item_agent_plan_read` with source `ai_workflows`, not tied to a run | W3 for the run id | | Workplan MRs vs other MRs | MR, review and pipeline tables | W1 and W3 | | Client split | Workflow `DUO_WORKFLOW_ENVIRONMENT` | W3 (F8 for IDE runs) | | Cost per run | Sum `token_usage_duo_agent_platform` by workflow id, or join `WK_FCT_DUO_WORKFLOW` on `SESSION_ID` | Nothing | ### Work #### W1 Link workflow runs to work items Add `issue_id`, `source_type` and `is_goal_workplan_prefixed` to the replicated `duo_workflows_workflows` table. The last column is an md5 match of the first 263 characters of `goal` against the workplan button's fixed prefix, so goal text never leaves Postgres. W5 sends `source_type` 15 from the workplan button, so new runs need no flag. The flag marks runs from before W5, when the button sent 7 like the header button. `issue_id` ties each run to a work item and its labels. - Repositories: two MRs. gitlab-data/extract!797 for the manifest, then gitlab-data/analytics!14647 for the dbt model. - Owner: data team. We draft the MRs. - Where: the extract manifest `gitlab_saas_postgres_pipeline/manifests/el_gitlab_dotcom_db_manifest_main.yaml`, then the dbt model for `GITLAB_DOTCOM_DUO_WORKFLOWS_WORKFLOWS`. - Status: draft MRs gitlab-data/extract!797 and gitlab-data/analytics!14647. Rollout order: extract MR merges, backfill and `_TEMP` swap run, the data team runs `dbt run --full-refresh -s gitlab_dotcom_duo_workflows_workflows_dedupe_source`, then the analytics MR merges. #### W2 Replicate the plan tables Replicate `work_item_agent_plans`, `work_item_decisions`, `work_item_decision_options` and `duo_workflows_workflow_work_items`. Plan content stays out because it lives in object storage. This gives plans per work item, the current readiness score, and decisions per work item. `workflow_id` is kept for a later agent vs human split, but the application does not populate it today. - Repositories: two MRs. gitlab-data/extract!798 for the manifest, then gitlab-data/analytics!14648 for the dbt models. - Owner: data team. We draft the MRs. - Where: one extract MR in the same manifest, plus one dbt model. `work_item_agent_plans` is at `db/structure.sql:35135-35144` (`work_item_id`, `namespace_id`, `created_at`, `updated_at`, `readiness_score`, `ai_planning_enabled`). - Status: draft MRs gitlab-data/extract!798 and gitlab-data/analytics!14648. #### W3 Work item id and run id on plan events Add `work_item_id` and `workflow_id` to the four `work_item_agent_plan_*` events (create, update, destroy, read) through `TrackingService.plan_event_properties`. The run id comes from the `X-Gitlab-Duo-Workflow-Id` header, read as ApplicationContext `duo_workflow_id`. The header is client-supplied, so the value is for analytics only. This gives revisions per work item and per-run plan reads. Count `work_item_agent_plan_update` by `source`: `ai_workflows` is the agent, `internal` is the browser, `api` is a PAT, glab or MCP. - Repository: one MR in gitlab-org/gitlab, !259076. - Owner: Plan team, one monolith MR. - Where: `lib/gitlab/work_items/instrumentation/tracking_service.rb`, the four event YAMLs, `ee/app/services/work_items/callbacks/agent_plan.rb` and `ee/app/graphql/types/work_items/widgets/agent_plan_type.rb`. - Status: MR !259076, open and ready for review. #### W4 Readiness score viewed event A new internal event fired from `work_item_confidence_score.vue` when the score popover or panel is shown. It is the only way to count score views. - Repository: one MR in gitlab-org/gitlab, !259077. - Owner: Plan team, one small frontend MR. - Where: `ee/app/assets/javascripts/work_items/components/ai_widget/work_item_confidence_score.vue`, plus the event YAML. - Status: MR !259077, merged 2026-10-02. Few scores are written today, because master runs workplan 1.0.0, which does not score. #### W5 Workplan Implement starts get their own source Add the client source type `work_item_plan_to_merge_request` and send it from the two workplan Implement buttons. They used to send `work_item_to_merge_request`, the same as the work item header button. The existing `agent_platform_session_created` event then carries the flow in `label` and the surface in `source`, and the value also lands in `duo_workflows_workflows.source_type`, which W1 replicates. New runs no longer need the goal-prefix match. We do not reuse `start_duo_workflow_execution`. The GitLab language server fires it for IDE and Duo CLI runs, and two agent_execution Service Ping metrics count it. - Repository: two MRs in gitlab-org/gitlab, backend !259107, then frontend !259421 (stacked on the !259107 branch). - Owner: Plan team. Needs agent_execution review, since they own the workflow model and the event. - Where: `ee/app/models/ai/duo_workflows/workflow.rb`, the `DuoWorkflowSourceType` GraphQL enum, `duo_work_item_to_mr_action.vue` and the two `implementation_widget/workplan_*.vue` components. - Status: split for multi-version safety. Backend !259107 (source type 15) merges first. Frontend !259421 (draft, stacked on the !259107 branch) sends it. Older Rails nodes reject unknown `source` values with a 400, so the frontend needs !259107 in a previous release, or an off-by-default feature flag, to stay safe for self-managed zero-downtime upgrades. Counts successful starts only, not failed clicks. ### Order, MRs and epic W1 to W5 can all start now, and none blocks another, except that the W5 frontend MR waits for the W5 backend deploy. W3 data is only useful per work item once W1 or W2 lands. W4 matters once workplan 1.1.0 or `readiness_score/v1` is enabled. - W1: one MR in gitlab-data/extract and one in gitlab-data/analytics, with a full refresh the data team runs between them. - W2: one MR in gitlab-data/extract and one in gitlab-data/analytics. - W3: one MR in gitlab-org/gitlab. - W4: one MR in gitlab-org/gitlab. - W5: two MRs in gitlab-org/gitlab, backend !259107 then frontend !259421. Each new event or extra needs Analytics Instrumentation review. Track it as one epic with five child issues, W1 to W5, plus one more issue that lists F1 to F11. Keep the follow-ups out of the epic's exit criteria. ### Nice-to-have follow-ups <details> <summary>Details</summary> None of these block the metrics above. Each one sharpens a number or adds a detail. | ID | What | What it adds | | --- | --- | --- | | F1 | Extract `trigger_source` and other workflows columns (`execution_mode`, `trigger_event_type`, `sync_type`, `tool_call_approvals`) | Keeps system runs out of adoption counts. Enables the async and sync split. | | F2 | Expose `merge_requests_closing_issues` in PROD | Counts MRs a human opened by hand after reading a plan. Today's proxy is note cross-references, which cover about 8% of developer-flow MRs. | | F3 | Flatten the new event extras into columns | Faster queries only. | | F4 | Click event on the Implement button | Failed clicks only. W5 already covers successful starts per person. | | F5 | Plan viewed event | Real plan views. Today `_read` fires on every page load of a work item with a plan. | | F6 | Backend readiness score event (`work_item_agent_plan_score_update`) | Score history over time. | | F7 | An `mcp` value for `source` | Separates MCP clients from glab and scripts. | | F8 | IDE runs send the run id | Plan-read rates for IDE developer runs. This is an Editor Extensions change. | | F9 | Gate name on pause events | Which workplan gate waits on a human. Only 1 pause in 18 workplan runs from 2026-09-25 to 2026-09-30. | | F10 | Track planner decisions: set `response_schema_tracking` on the workplan planner component | Decisions the planner records. YAML only. | | F11 | Plan-in-checkpoint study | A one-off Postgres check that a run acted on the plan. No MR. | </details> ### Query notes and open questions Query notes: - Join `WK_FCT_DUO_WORKFLOW` on `TRY_TO_NUMBER(SESSION_ID) = DUO_WORKFLOW_ID`, not on `WORKFLOW_ID`, which is a 38-character key. - Filter with `RAW_DUO_WORKFLOW_REQUEST_CATEGORY = 'workplan'`. The display column says `Unknown`. - For flow name and version, run `LATERAL FLATTEN` on `CONTEXTS`, then pick the `ai_context` entry (schema 1-0-2 since 2026-09-25). - Event extras are in `GSC_EXTRA` as JSON, for example `gsc_extra:source`. Open questions: - Why do 41% of `developer/v1` runs have no `WK_FCT_DUO_WORKFLOW` row? - When will workplan 1.1.0 or `readiness_score/v1` be enabled on gitlab.com, so that W4 has data?