Add MR count measurements to DuoWorkflows aggregation engine
What does this MR do and why?
Add MR count measurements to DuoWorkflows aggregation engine
Engineering leaders evaluating the GitLab Duo Agent Platform want to know how many merge requests it created and how many of those actually merged, without stitching the numbers together outside GitLab. The DAP Impact dashboard needs these counts broken down by group, project, flow type, and time window on both its Group comparison and Activities tabs. Before this change no single number existed for this, and a merged count was not even possible until the underlying table tracked later merge request outcomes.
This MR adds three measurements to the DuoWorkflows aggregation engine: created, merged, and closed merge request counts. Each one reads the length of an array column on the ClickHouse enriched table. Every measurement automatically expands into min, max, mean, quantile, and sum metrics, exposed in GraphQL as nested fields createdMrCount, mergedMrCount, and closedMrCount under the duoWorkflows aggregation. Dashboard totals come from the sum fields.
The merged and closed counts depend on a materialized view that refreshes every 12 hours and a reconciler that only rechecks flows from the last 30 days, so those two counts can lag real merge request state.
Resolves https://gitlab.com/gitlab-org/gitlab/-/work_items/617064
References
- Work item: https://gitlab.com/gitlab-org/gitlab/-/work_items/617064
- Epic, DAP Impact Dashboard v1: https://gitlab.com/groups/gitlab-org/-/work_items/23225
Screenshots or screen recordings
| Before | After |
|---|---|
How to set up and validate locally
- Start ClickHouse:
gdk start clickhouse
2. In a Rails console, enable the feature flag:
Feature.enable(:dap_impact_v1)
3. In the same console, insert a sample flow with two created merge requests, one of which later merged. Replace the placeholders with real IDs. The traversal path is the organization ID followed by the group ID.
ClickHouse::Client.execute(<<~SQL, :main) INSERT INTO duo_workflows_workflows_enriched (id, user_id, project_id, workflow_definition, status, created_at, updated_at, traversal_path, credits_used, created_merge_request_ids, later_merged_merge_request_ids, later_closed_merge_request_ids, _version) VALUES (1, <user_id>, <project_id>, 'software_development', 3, now(), now(), '<org_id>/<group_id>/', 1.5, [101, 102], [101], [], now()) SQL
4. Run this query in the GraphQL explorer at /-/graphql-explorer, using the full path of that group:
query { group(fullPath: "") { analytics { duoWorkflows { aggregated { nodes { totalCount createdMrCount { sum } mergedMrCount { sum } closedMrCount { sum } } } } } } }
5. Confirm the response shows totalCount 1, createdMrCount.sum 2, mergedMrCount.sum 1, and closedMrCount.sum 0.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Related to #617064