Draft: Split Duo Workflow credits by the model that spent them

Jatin Pariharrequested to merge
jparihar/credits-by-model-630075 into master

What does this MR do and why?

The DAP Impact "credits by model" chart attributes 100% of a session's credits to whichever model made the most LLM calls. A session using five models shows as one bar. This stores the real split and exposes it.

Three commits, reviewable in order:

  1. ClickHouse columns. models and model_credits as positional arrays on duo_workflow_session_enrichments, carried into duo_workflows_workflows_enriched.
  2. Ingest. Records the split CustomersDot now reports, next to the session total.
  3. Aggregation engine. A new DuoWorkflowModelCredits engine returning credits at (flow, model) grain.

Why arrays on the session row rather than per-model rows

A table keyed (workflow_id, model_used) is the obvious shape and does not work. Rows sharing a sort key and version in a ReplacingMergeTree collapse to one arbitrary survivor — verified locally: three rows for one session, three models, one row kept, before any merge was forced. Separately, a session's model set shrinks as events age past the 90-day fetch window, so per-model rows would linger as stale totals with nothing for the writer to tombstone them against.

Arrays on the existing session row avoid both, because whole-row replacement already handles a shrinking model set there exactly as it does for credits_used. Added with ALTER TABLE ... ADD COLUMN, so no enrichment data is dropped.

Both MVs that append to duo_workflows_workflows_enriched carry the new columns. The outcomes MV re-emits whole rows it read back, so leaving it alone would let it overwrite a session's split with the column default on merge.

Why a separate engine rather than a dimension on DuoWorkflows

Splitting a session across models multiplies rows, and credits_per_merged_mr — which now backs two Overview tiles — would keep a correct numerator while its denominator tripled for a three-model session. It cannot be patched on that one metric: dividing the MR count by the model count fixes it while expanded and breaks it while not.

The table is still shared. DuoWorkflows reads it exactly as before and never sees an arrayJoin; its specs pass unchanged, credits_per_merged_mr included. Only the new engine expands, at query time, and it exposes credits alone — flow and user counts are meaningless once a session is split.

models and model_credits are positional, so one arrayJoin over their arrayZip expands them together; two separate arrayJoins would cross join the models against the amounts. There is no model filter, because filters apply before the split, where the expanded model does not exist yet.

Unattributed credits

A session CustomersDot cannot split books its credits under an empty model. The reading side expands these arrays, so a session absent from them would quietly shrink the charted total below what was actually spent. Sessions ingested before this ships keep empty arrays until the 15-minute cron re-ingests them, so historical data under-reports until it catches up.

References

Screenshots or screen recordings

No UI in this MR. Repointing the dashboard panels (dap_impact.yaml) and registering the GLQL source are follow-up work.

How to set up and validate locally

  1. bundle exec rake gitlab:clickhouse:migrate
  2. bundle exec rspec ee/spec/services/ai/duo_workflows/session_credits/ ee/spec/models/analytics/aggregation_engines/ ee/spec/requests/api/graphql/analytics/ ee/spec/lib/gitlab/subscription_portal/subscription_usage_client_spec.rb
  3. The regression that matters most is duo_workflows_spec.rb staying green — it proves the existing engine, and credits_per_merged_mr with it, is untouched.

MR acceptance checklist

  • Migrations applied and rolled back locally, then re-applied.
  • No changelog trailer: the whole duoWorkflows surface is behind the default-off dap_impact_v1 flag.
  • Wants a look from Application Security. The new engine's generated types needed granular token authorization, and I chose read_enterprise_ai_analytics with group and project boundaries because it is the only defined permission covering Duo analytics. The sibling DuoWorkflows engine has never made that choice — it sits in config/authz/graphql/authorization_todo.txt, which forbids new entries — so there was no precedent to copy. Please confirm the permission and boundaries are right.
  • Database review wanted for the two ClickHouse migrations.

Merge request reports