Tell the Data Analyst agent how to query the previous period

Rudy Cresporequested to merge
631885-analyze-with-duo-previous-period into master

What does this MR do and why?

Tells the Data Analyst agent how to answer previous-period questions and what to report after a failed query, and adds tooltips to the DAP Impact Spend panels so the agent (and users) see what each panel counts.

This is a follow-up to !259780 (merged), which added rules to the additional_instructions that ee/app/assets/javascripts/explore/analytics_dashboards/components/analyze_with_duo_button.vue sends to the agent, and started sending each panel's tooltip.description. An evaluation of the agent found two failures those rules do not cover, plus a gap in the DAP Impact dashboard.

This MR makes three changes:

  • Previous period (analyze_with_duo_button.vue)
    • The context sends date_range.previous (start and end dates) but never said what they were for. The agent used a previous-period metric such as previousPeriodUsersCount instead. That metric compares each created date bucket with the one before it, not with the previous date range, and it is rejected unless created is a dimension.
    • New instruction: "To compare a total with the previous period, run the same panel query again with the dates in date_range.previous."
    • New instruction: comparison metrics such as previousPeriodUsersCount and returningUsersCount compare each date bucket with the one before it, so they need a date dimension. To compare whole periods, use buckets as long as the date range, with origin set to the start of the previous period. This keeps retention questions, which need these metrics, working.
  • Numbers after a failed query (analyze_with_duo_button.vue)
    • After a query failed, the agent sometimes reported a figure that no query returned, or reused a number from a different metric and said it came from a panel.
    • New instruction: "If a query returns an error, fix it and run it again, or say that the number is not available. Only report numbers that a query in this conversation returned, or that you calculated from them, and do not reuse a number from a different metric, filter, or date range."
  • Spend tooltips (ee/lib/gitlab/analytics/dashboards/system/dap_impact.yaml)
    • Added tooltip.description to four Spend panels: "Credits by model, by capability", "Credits", "Chat", and "Credits over time".
    • "Credits" and "Chat" already had a short displayConfig.description inside their GLQL. The other two had nothing.
    • The wording follows the tooltips on the other views. For example, "Sessions over time" says "stacked by flow type", and "Sessions by tier and top capabilities" calls flow types capabilities.

There is no feature flag. The change is instruction and tooltip text only and is easy to revert.

References

How to set up and validate locally

  1. Open the DAP Impact dashboard, Overview view, and click "Analyze with Duo".

  2. Ask "How many active users were there in this period and in the previous period?". Check that both numbers come from the Active users query, run once with each period's dates, and not from a previous-period metric.

  3. Open the Spend view and hover each panel title to see the tooltips.

  4. On the Spend view, open "Analyze with Duo" and check that each panel's description reaches the agent (the dashboard context's panels[].description).

  5. Run the Jest spec:

    yarn jest ee/spec/frontend/explore/analytics_dashboards/components/analyze_with_duo_button_spec.js

After this deploys, the evaluation harness copies of both files need a sync. That is tracked separately and is not part of this MR.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code

Edited by Rudy Crespo

Merge request reports