Add the tool calls card, ClickHouse reader and GraphQL field to the AI Governance dashboard

What does this MR do and why?

Adds the "Top tools used by agents" card and a "Total number of tool calls" summary tile to the AI Governance dashboard, with the ClickHouse reader and GraphQL fields behind them. Both sit behind the ai_governance_tool_calls flag (wip, off by default).

This MR now includes the summary tile from Add AI tool calls summary tile to the AI Govern... (!256856 - merged) • Dheeraj Joshi • 19.5, which was reviewed separately and merged into this branch.

Card:

  • Ai::Governance::ClickHouseToolCallsService: top 10 tools by call count for a group or project, timeframe and agent class, from ai_audit_events in ClickHouse.
  • Ai::Governance::ToolCallsService: uses ClickHouse when Gitlab::ClickHouse.enabled_for_analytics? is on and returns an error otherwise. It owns the shared EVENTS and TOP_N constants.
  • aiGovernanceToolCalls on Group and Project (experiment): tool invocations by tool name for a timeframe and agent class, plus an approximate flag. Nullable, gated behind ai_governance_dashboard and ai_governance_tool_calls.
  • The card: one row per tool with its call count for the dashboard's date range and agent class, a note when counts are approximate, and a link to the Tool management tab. When the field is null (no ClickHouse) it shows its empty state.

Tile:

  • A toolCalls KPI on aiGovernanceMetrics with count, previousCount, trend and cumulativeTrend. The running total is seeded from the 30 days before the window, the same as sessions.
  • It counts every ai_tool_invoked invocation once, with the same dedup and tool-name rule as the card, so the tile total and the card agree.
  • Agent class is resolved through the workflow, the same way the card does it.
  • ClickHouse only. Without ClickHouse the field is null and the tile shows a dash.

The PostgreSQL reader and KPI for instances without ClickHouse follow in Add tool-call tally scopes and PostgreSQL tool ... (!256523 - closed) • Dheeraj Joshi • 19.6 and plug into the same services.

Example query
query {
  group(fullPath: "gitlab-org") {
    aiGovernanceToolCalls(timeframe: LAST_7_DAYS, agentClass: ALL) {
      approximate
      nodes {
        toolName
        count
      }
    }
  }
}

Screenshots

Tiles, last 30 days
kpi-row-light-30d
Card Dark mode Without ClickHouse
card-light-30d card-dark-30d card-empty
Tile, dark mode Tiles without ClickHouse
tile-dark kpi-row-no-clickhouse

How to validate locally

Needs ClickHouse enabled for analytics (Gitlab::ClickHouse.enabled_for_analytics? true) and audit events flowing into it.

  1. Enable the flags for a group: Feature.enable(:ai_governance_dashboard, group) and Feature.enable(:ai_governance_tool_calls, group).
  2. Run a few Duo agent sessions in a project of that group so ai_tool_invoked audit events land in ClickHouse, or insert rows into ai_audit_events and siphon_duo_workflows_workflows as ee/spec/services/ai/governance/click_house_tool_calls_service_spec.rb does. Add one row older than 30 days and one with an empty tool_name: neither should show.
  3. Open the group's Settings > GitLab Duo > Governance, Dashboard tab. The "Top tools used by agents" card lists the tools by call count.
  4. Switch the date range and the agent class filter. The card follows both.
  5. Click View tool management. It opens the Tool management tab.
  6. Check the "Total number of tool calls" tile, its delta badge and its trend. Confirm it follows the date range and the agent class filter.
  7. Query the API in GraphiQL with the example query above. The counts match the card.
  8. Query aiGovernanceMetrics { toolCalls { count previousCount trend } } and compare with the tile.
  9. Disable ClickHouse for analytics. The card field returns null and the card shows its empty state. The tile shows a dash.

ClickHouse review

ai_audit_events is a ReplacingMergeTree partitioned by month, ORDER BY (traversal_path, workflow_id, created_at, id). The query filters on traversal_path prefix and created_at, both in the sort key, and deduplicates on the full sort key before counting because ReplacingMergeTree collapses re-ingested rows only on merge. agent_type lives on siphon_duo_workflows_workflows (ORDER BY (traversal_path, created_at, id)), so the agent-class filter is a semi-join on workflow_id; it has no lower time bound because a session can start long before the events it emits into the window.

The tile's KPI queries (totals over the current and previous window, per-bucket trend, and a baseline for the running total) reuse the same dedup subquery and agent-class filter as the card's query, in ClickHouseMetricsService.

I don't have access to production ClickHouse to attach a plan. The spec runs the query against a real ClickHouse server; if a reviewer can run EXPLAIN indexes=1 on gitlab-org, I'd welcome it.

Query, group scope, :internal_dap (:all drops the workflow_id IN block)
SELECT
  JSONExtractString(details, 'tool_name') AS tool_name,
  count() AS calls
FROM (
  SELECT traversal_path, workflow_id, created_at, id, any(details) AS details
  FROM ai_audit_events
  WHERE startsWith(traversal_path, {traversal_path:String})
    AND created_at >= {from:DateTime64(6, 'UTC')}
    AND created_at < {to:DateTime64(6, 'UTC')}
    AND event_name IN {events:Array(String)}
    AND workflow_id IN (
      SELECT id FROM (
        SELECT id,
          argMax(agent_type, _siphon_replicated_at) AS agent_type,
          argMax(_siphon_deleted, _siphon_replicated_at) AS deleted
        FROM siphon_duo_workflows_workflows
        WHERE startsWith(traversal_path, {traversal_path:String})
          AND created_at < {to:DateTime64(6, 'UTC')}
        GROUP BY traversal_path, created_at, id
      )
      WHERE deleted = false
        AND agent_type IS NULL
    )
  GROUP BY traversal_path, workflow_id, created_at, id
)
WHERE tool_name != ''
GROUP BY tool_name
ORDER BY calls DESC, tool_name ASC
LIMIT {limit:UInt64}

Placeholders: traversal_path is the namespace's traversal_path(with_organization: true), for example 1/9970/; events is ['ai_tool_invoked']; from/to are the dashboard window, upper bound exclusive; limit is 10. :external uses agent_type IS NOT NULL.

Specs

  • ee/spec/services/ai/governance/click_house_tool_calls_service_spec.rb (runs against ClickHouse)
  • ee/spec/services/ai/governance/tool_calls_service_spec.rb
  • ee/spec/services/ai/governance/click_house_metrics_service_spec.rb
  • ee/spec/requests/api/graphql/ai/governance/tool_calls_spec.rb (runs against ClickHouse)
  • ee/spec/requests/api/graphql/ai/governance/metrics_tool_calls_spec.rb
  • ee/spec/requests/api/graphql/ai/governance/metrics_spec.rb
  • ee/spec/frontend/ai/governance/components/dashboard/cards/tool_calls_card_spec.js
  • ee/spec/frontend/ai/governance/components/dashboard/dashboard_app_spec.js

Follow-ups

Agent class is resolved through the workflow today. When ai_governance_sessions lands (#630095), this moves to the session's source.

Edited by Dheeraj Joshi

Merge request reports

Loading
Loading