Add credits freshness caveat to DAP Impact tooltips

Chandra Saripakarequested to merge
632230-dap-impact-credit-data-lag into master

What does this MR do and why?

Adds one closing sentence to the tooltip description of every credits panel in the DAP Impact system dashboard (ee/lib/gitlab/analytics/dashboards/system/dap_impact.yaml):

Credits are fetched from billing after a session is billed and can lag behind the session counts, so recent totals can understate spend.

Panels changed:

  • Overview: "Credits consumed", "Where credits went"
  • Adoption: "Group and project comparison" (it lists credits per project)
  • Spend: "Credits by model, by capability", "Credits", "Chat", "Credits over time"

Not changed: the Spend "Credits by model" panel (it has no tooltip) and the on-tile subtitles (displayConfig.description).

Since !259780 (merged), the Data Analyst agent receives each panel's tooltip description as context when opened from the dashboard. The sentence therefore reaches people hovering the panel and the agent answering about it. In the UAT harness (#631885) the agent described missing credits as "about a week behind", which was a confident guess.

The sentence is worded to be true in both states below and has no number that could go stale.

  • The duo_workflow_session_credits_ingestion rollout (#607369) is blocked with CustomersDot prerequisites unchecked. The flag was switched off after the 2026-09-05 global enable overloaded CustomersDot. The v2 pull path (#629767, #629769) is still in review. With the flag off for a namespace, the cron scans nothing, so credits on the Spend tab stop at the last ingested window. That looks like lag but is paused ingestion.
  • When ingestion runs, the designed gap is the 2-hour settle horizon plus up to one 15-minute tick.

Related to #632230. This MR does not close it: the measurement part is still open. Two checks need someone with access: the chatops flag actor list, and a credits-by-day query run as a viewer with read_agent_artifacts.

A visible freshness note in the Spend tab header needs design and is filed as #632859.

How to test

  1. Open the DAP Impact dashboard.
  2. Hover the info icon on any credits panel on the Overview or Spend tab and read the sentence.
  3. Click Analyze with Duo and confirm the request's panels[].description carries the sentence.

Tests

ee/spec/lib/gitlab/analytics/dashboards/system/dap_impact_spec.rb gains an example that lists every panel whose tooltip mentions credits and asserts each one ends with the caveat sentence. It fails on master's YAML and passes here. system_dashboard_spec.rb and dashboard_constraints_spec.rb still pass (26 examples).

Plan

Plan for #632230 (this MR is Phase 2, Task 4)

Confirm credit data lag and tell the Data Analyst agent — plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Establish, with production evidence, how late Duo credit data appears on the DAP Impact Spend tab, write the finding on the issue, and make the lag visible to the Data Analyst agent and to readers through the Spend panel descriptions.

Architecture: Two phases. Phase 1 is measurement with no code: feature flag state, Sidekiq cursor_lag_seconds and back-off warnings, and a credits-by-day comparison for a known namespace. Phase 2 is a YAML-only change: the Spend panel tooltip descriptions carry the measured lag and the null-means-not-yet-billed rule, because MR 259780 made panel descriptions part of the agent's context. A visible UI banner needs design and is split into its own issue.

Tech Stack: Kibana (Sidekiq logs), ChatOps feature flag query, GitLab GraphQL, dashboard YAML.

Spec: https://gitlab.com/gitlab-org/gitlab/-/work_items/632230. Follow-up of 631885.

Global Constraints

  • Nothing in Phase 1 writes anywhere except the issue.
  • Phase 2 edits only ee/lib/gitlab/analytics/dashboards/system/dap_impact.yaml. No new agent instruction field is added unless the measured lag cannot be expressed in panel descriptions.
  • Numbers in descriptions are round and conservative ("about 3 hours", "up to a day"), never a precise figure that goes stale.
  • Draft MR title carries [skip ci] until ready.

Review Focus

  1. A viewer without read_agent_artifacts sees NULL credits regardless of lag; the description must not make them think the data is merely late. Pinned by wording in Task 4 and the existing "no access" tile state.
  2. A session that billed and then moved to failed before the cron scanned it is missing, not late (#617273). The description must say "not yet billed or not fetched", not promise it will arrive. Pinned in Task 4.
  3. The agent reads descriptions verbatim; a description longer than two sentences crowds the prompt. Pinned by the length check in Task 4.
  4. If the flag is off for the viewer's namespace there is no credit data at all; Task 1 must record the flag state before any lag claim is made.
  5. credits_used defaults to 0 in the enriched table only for rows the ingest wrote; a missing enrichment row reads NULL through the engine. Task 2's comparison must distinguish NULL from 0.

Phase 1 — Measure

Task 1: Flag state and rollout status

Files: none. Output goes to the issue.

  • Step 1: Read the rollout issue
glab api projects/278964/issues/607369 | jq -r '.state, .labels[], .description[0:1500]'
glab api 'projects/278964/issues/607369/notes?sort=desc&per_page=5' | jq -r '.[] | select(.system==false) | "[\(.author.username) \(.created_at[:10])] \(.body[0:400])"'

Record: whether the flag is enabled for gitlab-org, for a percentage, or off, and since when.

  • Step 2: Confirm the live flag state

The user runs in Slack: /chatops run feature get duo_workflow_session_credits_ingestion. Paste the actor list into the issue note drafted in Task 3.

Task 2: Worker lag and back-off from logs, and a credits-by-day comparison

Files: none.

  • Step 1: Kibana, Sidekiq index, last 7 days

Query for the cron's end-of-run metadata:

json.class: "Ai::DuoWorkflows::FetchSessionCreditsCronWorker" AND json.job_status: "done"

Visualise json.extra.ai_duo_workflows_fetch_session_credits_cron_worker.cursor_lag_seconds (max per hour) and ...stop_reason (terms). The field prefix follows log_extra_metadata_on_done; confirm the exact key in one document first.

Back-off warnings:

json.class: "Ai::DuoWorkflows::FetchSessionCreditsCronWorker" AND json.message: "Skipping session credits cycle while backing off from CustomersDot"

Record: max cursor_lag_seconds in the window, how often stop_reason is budget or backoff, and the error_type of back-offs.

  • Step 2: Credits by session day for gitlab-org

Run as a user with read_agent_artifacts on gitlab-org (the author does not have it; ask ghosh-abhinaba or rcrespo3):

query { group(fullPath: "gitlab-org") { analytics { duoWorkflows(createdAtFrom: "<today-14d>T00:00:00Z", createdAtTo: "<today>T23:59:00Z") { aggregated { nodes { dimensions { createdAt(granularity: "daily") } totalCount creditsUsed { sum } } } } } } }

Record the first day, counting back from today, where creditsUsed.sum stops being NULL and roughly matches the run-rate of earlier days. The gap between that day and today is the user-visible lag. Repeat once filtered to status: ["finished"] to separate missing-because-failed from late.

  • Step 3: Compare one session end to end

Pick one finished gitlab-org session from yesterday (id from the same query with dimensions { } removed and first: 1), then in a Rails console on a production console host:

ClickHouse::Client.select("SELECT workflow_id, credits_used, updated_at FROM duo_workflow_session_enrichments WHERE workflow_id = <id>", :main)

Record whether the row exists and its updated_at relative to the session's updated_at. If it is missing, check the session's status history: a session that billed and then failed is #617273, not lag.

Task 3: Write the finding on the issue

Files: none.

  • Step 1: Post one note with this shape
Measured on <date> for gitlab-org.

- Flag: <enabled for gitlab-org since <date> | percentage N | off>. Source: rollout issue 607369 and chatops.
- Worker: max cursor_lag_seconds <n> over 7 days, stop_reason mostly <x>, <k> back-off warnings (<error types>).
- Credits by day: non-null from <day>, so the visible lag is about <n> hours/days. Filtered to finished sessions: <same | different>.
- One session: enrichment row <present after <n> h | missing>, which is <lag | #617273>.

Conclusion: <the week-long figure in 631885 is / is not reproduced>; the mechanism is <worker budget | CustomersDot delay | flag coverage | missing rows>.

Keep it to those lines. No details block.

  • Step 2: Decide the branch

  • Lag confirmed at hours: go to Phase 2 with "about N hours".

  • Lag confirmed at days: go to Phase 2 with "up to N days" and open a separate issue for the worker or CustomersDot cause, linked from this one.

  • No lag, only missing rows: close this issue as a duplicate of #617273 after the Phase 2 wording change, since "not yet billed or not fetched" still helps the agent.

Phase 2 — Tell readers and the agent

Task 4: Spend panel descriptions

Files:

  • Modify: ee/lib/gitlab/analytics/dashboards/system/dap_impact.yaml, Spend view (search - title: Spend), tooltips of Credits by model, Credits by model, by capability, Credits, Chat, Credits over time, and the Overview Credits consumed tile.
  • Test: ee/spec/models/analytics/custom_dashboards/system_dashboard_spec.rb, ee/spec/models/analytics/custom_dashboards/dashboard_constraints_spec.rb.

Interfaces:

  • Consumes: the lag figure from Task 3.

  • Step 1: Write the sentence once

The same closing sentence goes on every credits panel tooltip, so the agent sees it whichever panel it reads:

Credits arrive from billing about <N> after a session ends; a session not yet billed or not yet fetched has no credits and is shown as no data, not zero.

Replace <N> with the Task 3 figure ("3 hours", "a day").

  • Step 2: Apply it

For each listed panel, append the sentence to tooltip.description after the existing text. Keep each description to two sentences. Example for Credits:

    tooltip:
      description: Total credits consumed by GitLab Duo Agent Platform (DAP) sessions in the selected date range. Credits arrive from billing about <N> after a session ends; a session not yet billed or not yet fetched has no credits and is shown as no data, not zero.

Do not touch displayConfig.description (the on-tile subtitle); that text is for the reader at a glance and the lag note would crowd a stat tile.

  • Step 3: Length check
awk '/- title: Spend/,0' ee/lib/gitlab/analytics/dashboards/system/dap_impact.yaml | grep "description:" | awk '{ print length, $0 }' | sort -rn | head -3

Expected: no tooltip description over about 320 characters.

  • Step 4: Run the dashboard specs
./tmp/run.sh bundle exec rspec ee/spec/models/analytics/custom_dashboards/system_dashboard_spec.rb ee/spec/models/analytics/custom_dashboards/dashboard_constraints_spec.rb

Expected: pass.

  • Step 5: Check the agent sees it

Render the Spend tab on GDK with production data (Playwright recipe in the gdk-has-no-license-for-engines memory) and click Analyze with Duo; confirm the request payload's panels[].description carries the sentence. No screenshot upload without the user's yes.

  • Step 6: Commit and open the MR
git add ee/lib/gitlab/analytics/dashboards/system/dap_impact.yaml
git commit -m "State the credit data lag on DAP Impact credits panels"

Title [skip ci] State the credit data lag on DAP Impact credits panels. Body: the measured figure, the Task 3 note link, that the on-tile subtitles are unchanged, and Closes #632230 only if Phase 1 confirmed lag at hours; otherwise Related to #632230 and the follow-up issue.

Task 5: Visible lag note in the UI (design needed, separate issue)

Files: none in this MR.

  • Step 1: Open an issue titled "DAP Impact Spend: show how recent the credit data is", labels from this issue plus UX, describing the measured lag, pointing at the Spend tab header as the candidate placement, and linking 632230. Assign to no one; mention the dashboard designer who owns 628029.

Review note: the earlier wording promised that unfetched credits show as no data. The enriched table stores credits_used as Float64 DEFAULT 0 and the join fills unmatched sessions with 0, so the panels show 0, not a gap. The sentence now only says that totals can understate spend.

Edited by Chandra Saripaka

Merge request reports