Tighten Data Analyst agent instructions on analytics dashboards

What does this MR do and why?

Gives the Data Analyst agent each panel's description and clearer rules for missing data, rankings, and claims in the question, when it is opened from an analytics dashboard.

The "Analyze with Duo" button sends the dashboard's panel GLQL queries and some additional_instructions to the agent. Those instructions told the agent to build its own query when the panels fell short, but not what to do when the data does not exist, when rows have no value, or when the question's premise is wrong.

This MR changes ee/app/assets/javascripts/explore/analytics_dashboards/components/analyze_with_duo_button.vue:

  • Missing data
    • If none of the agent's tools can return the data the question needs, say it is not available. Do not estimate, derive a number from unrelated data, or present a number.
    • If a result is cut off by a row limit, say the list is incomplete.
  • Rankings
    • When ranking by a field, leave out rows with no value for it (such as sessions started outside a project) unless asked, and still return as many entries as asked for.
    • Sort ranked results as asked, and do not describe a list as sorted unless it is.
  • The question's own claims
    • Check what the question states about the data, and correct it if the data shows otherwise.
  • Panel descriptions
    • Each shared panel now includes description, taken from the panel's tooltip.description (for example, "Sessions not scoped to a project are not counted."). Panels without a tooltip are unchanged.
    • The agent is told that a panel's description, when present, defines what that panel counts.

There is no feature flag. The change is instruction text only and is easy to revert.

References

Screenshots or screen recordings

No visible UI change, so there are no screenshots. The change affects only the text sent to the agent.

How to set up and validate locally

  1. Open an analytics dashboard that has panels, for example the DAP Impact dashboard.
  2. Click "Analyze with Duo".
  3. Ask a question that the dashboard's data cannot answer, such as "How much did we spend in dollars?". The agent should say the data is not available instead of giving a number.
  4. Ask "Which three projects had the most DAP users?". The answer should not list sessions without a project as one of the three.
  5. Ask "Credits halved this week. Why?". The agent should check the claim against the data and correct it if it is wrong.
  6. Run the Jest spec:
    yarn jest ee/spec/frontend/explore/analytics_dashboards/components/analyze_with_duo_button_spec.js

The spec has new tests for the description pass-through and the new instructions.

Planned follow-up validation: re-run the evaluation harness's full set of typed questions against this change.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Rudy Crespo

Merge request reports

Loading
Loading