Add agent class segmentation to AI governance metrics

Andrew Jungrequested to merge
603776-ai-governance-metrics-agent-class into master

What does this MR do and why?

Adds an agentClass argument to the aiGovernanceMetrics GraphQL field (on both Group and Project) so the AI Governance Dashboard can segment its KPI tiles by agent class, and redefines the AI Agents KPI from distinct workflow definitions to distinct active agent instances.

This delivers https://gitlab.com/gitlab-org/gitlab/-/work_items/606071 in full. It targets master: the GraphQL API (!244211 (merged)) and the agent_type column (!241920 (merged)) have both merged. Everything is behind the default-off ai_governance_dashboard flag (wip), and the parent field is marked as an experiment.

Earlier revisions of this MR shipped a placeholder that returned zeros for EXTERNAL because agent_type was not yet on master. That is no longer needed, so the placeholder is gone and this MR ships the real predicates directly.

1. agentClass argument

New AiGovernanceAgentClass enum, applied identically on the PostgreSQL and ClickHouse paths, to totals, previous-window totals and every trend bucket:

Value Predicate
ALL (default) no predicate
INTERNAL_DAP agent_type IS NULL
EXTERNAL agent_type IS NOT NULL

chat sessions remain excluded from every class via MetricsService::EXCLUDED_DEFINITIONS.

Known limitation: INTERNAL_DAP is agent_type IS NULL, which is a catch-all rather than a positive predicate, so it absorbs anything no external producer wrote. In practice the absorbed population is legitimately internal: catalog-triggered runs (about 89,500 sessions per 30 days, 86,688 flows plus 2,820 user agents) execute inside DAP. The only genuinely external slice under it is third-party catalog agents, 11 sessions per 30 days, which does not justify a class of its own. Measured in #607564 (closed), where a CATALOG class was prototyped and retracted (!247948 (closed)). Read INTERNAL_DAP as "not an external agent session"; the enum is additive if a real population appears later.

2. Agents KPI: distinct active instances

workflow_definition counts "how many distinct flow types ran", not "how many agents are active", and it is meaningless for external sessions. The KPI now counts distinct active agent instances with a single key expression:

(user_id, project_id, namespace_id, agent_type, agent_identity_id,
 CASE WHEN agent_type IS NULL THEN
   CASE environment WHEN 1 THEN 4 WHEN 2 THEN 5 ELSE environment END
 END)

ClickHouse mirrors it with if(agent_type IS NULL, transform(environment, [1, 2], [4, 5], environment), NULL). Both environment mappings are derived from ENVIRONMENTS_DEPRECATIONS and Workflow.environments, so the two paths cannot drift.

One expression covers both classes, because agent_type IS NULL tells them apart:

Class Effective key One instance is
INTERNAL_DAP (user, container, normalized environment) one person on one Duo surface (chat, ambient) in one project or namespace
EXTERNAL (user, project, agent_type, agent_identity_id) one person running one agent product from one machine against one project

Internal DAP sessions have agent_type IS NULL and never carry an identity, so their key collapses to (user, container, normalized environment), where the container is the project or, for namespace-attached sessions, the namespace.

Internal numbers do move, downward, and that is the point. ide was renamed to chat and web to ambient in 18.6. ENVIRONMENTS_DEPRECATIONS records those as reason: :renamed, and both spellings have live writers in different repos today:

Value Written by
ide (1) gitlab-lsp, AGENT_IDE_ENVIRONMENT = 'ide' in workflow_rails_service.ts, on every IDE session
web (2) Ai::Catalog::ExecuteWorkflowService (WORKFLOW_ENVIRONMENT = 'web') and Ai::FlowTriggers::RunService
ambient (5) Duo Workflow Service start_flow tool, which POSTs "environment": "ambient"

web and ambient are the same product under the rename, so a project where flow triggers fire and agents call start_flow counts one person twice. Neither value is excluded from the counted set, since only workflow_definition = 'chat' is dropped. Normalizing collapses each renamed pair.

Verified locally with one session per spelling for the same user and project, four sessions across two products:

Agent instances
Raw environment (before) 4
Normalized (this MR) 2

Identical on both read paths, and the same 4 to 2 collapse is asserted in a spec on each. Reverting the normalization fails those specs with expected: 2, got: 4 on PostgreSQL and expected: 4, got: 6 on ClickHouse.

namespace_id is in the key for the mirror-image reason. Exactly one of project_id/namespace_id is set per row (the num_nonnulls check constraint), and namespace-attached sessions carry project_id NULL — without the column, one user's sessions across different namespaces in a hierarchy collapse into a single instance. Measured on production (2026-08-04): 84 user×root pairs in 30 days. For project-attached rows the column is NULL on both sides of the tuple and changes nothing.

External sessions all carry the same environment, so keying them on environment would collapse every external agent in a project onto one instance. agent_type splits them by product. agent_identity_id then splits them by machine: an identity is unique on (user_id, project_id, agent_type, machine_fingerprint), and the fingerprint is a SHA-256 of the platform machine ID, so one person running Claude Code from a laptop and from a CI runner counts as two instances rather than one. That reads the way the tile is labelled, an active fleet size rather than a product list.

Sessions with no identity fall back to per-product counting, which is also what happens if an identity is deleted later (the FK nullifies). NULL is a value inside the key tuple for both COUNT(DISTINCT ...) and uniqExact, so those sessions are still counted, they just group together.

The two key spaces stay disjoint, so ALL is exactly INTERNAL_DAP + EXTERNAL, which is asserted in a spec.

Trend semantics. Each trend bucket is computed independently, so a distinct-count KPI does not sum to its total: an agent active on three days contributes three points to agents.trend and one to agents.count. Sessions do sum. That asymmetry is now stated in the trend field description and locked by a spec, since the frontend renders value and chartData in the same tile.

The external count source was previously listed as an open question in #606071. It is settled here: distinct (user_id, project_id, agent_type, agent_identity_id) over external sessions, on both read paths. The count comes off the session row rather than out of ai_agent_identities, for two reasons. That table is not replicated to ClickHouse (there is no siphon_ai_agent_identities), so counting rows in it would only work on one of the two read paths. And an identity that has registered but never opened a session is not an active agent, which is what this KPI is for.

Taxonomy note: the design in #591226 asks for three classes, internal, external and guest. This MR ships two, and the mapping between them is unresolved. Be aware that "external agent" already has two published meanings: the GAIG design doc (handbook MR 20111) uses it for locally-run Claude Code and OpenCode, which is what EXTERNAL returns here, while doc/user/duo_agent_platform/agents/external.md has used it since 18.3 for AI Catalog agents, which run inside DAP and therefore fall into INTERNAL_DAP today. Enum values are additive, so further classes can be added without breaking the contract. Tracked in #606071.

Database notes

No migration. agent_identity_id already exists on duo_workflows_workflows and in the siphoned ClickHouse table. This MR changes only the aggregate expression: the row set, the in_namespace_hierarchy UNION scoping and the driving indexes are all unchanged from the merged aggregation MR.

Plans were captured against the final six-column, normalized-environment key on gitlab-production-main, session 54273 (2026-08-04). These are warm second runs; the first execution on a fresh clone is dominated by cold I/O and is not representative (the 7-day totals query first ran in 1,543 ms with 2,249 blocks read, then 41.8 ms with zero).

Query Time Buffers Plan
Group, totals (7d) 41.8 ms 17,387 https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/54273/commands/156903
Group, daily buckets (7d) 35.0 ms 14,993 https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/54273/commands/156905
Group, totals (30d) 83.3 ms 31,053 https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/54273/commands/156909
Group, daily buckets (30d) 76.2 ms 23,483 https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/54273/commands/156911
Project, totals (7d) 0.4 ms 119 https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/54273/commands/156913
Project, daily buckets (7d) 0.2 ms 19 https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/54273/commands/156915

Notes for the database reviewer:

  • No sequential scans. Both UNION branches use existing indexes, index_duo_workflows_workflows_project_environment_created_at for the project-attached rows and index_duo_workflows_workflows_on_namespace_id_created_at for the namespace-attached ones. No new index is proposed.
  • The default timeframe is 7 days, and one request runs a totals query plus a bucket query. So the default group path is roughly 77 ms and 32k buffers combined, and the 30-day group path roughly 160 ms and 55k buffers.
  • The tuple aggregate is the only part this MR adds, and it is CPU-only: the six-column key's sole visible footprint in the plans is a wider tuple feeding the Append (66 → 74 bytes per row); the scan shape, indexes and buffer profile are unchanged from the pre-MR aggregate.
  • The scope picker anchors on the most recent active project, so this run sampled a mid-size root: 316 groups and 2,693 projects. An earlier capture of the same query shapes on a larger root (733 groups, 3,655 projects, session 54160) showed the identical plan shape at 51.6/138.9 ms, so both the shape and its scaling with hierarchy size are documented. Group cost scales with project count because the project branch performs one index lookup per project.
  • Row estimates in these plans are unreliable by construction, not because of the change. The window bound arrives through an InitPlan ((SELECT ref_ts FROM scope)), so the planner falls back to default selectivity, which is why both group totals plans carry the same cost estimate despite covering different windows. Production passes literal timestamps.
  • Worth stating plainly: each 30-day group query is now under 100 ms on this sample, but one request runs two of them (~160 ms combined), and the larger-root capture put a single totals query at 138.9 ms. The whole field is behind the default-off ai_governance_dashboard flag, and this shape is inherited rather than introduced, but it is the number to watch when the flag is rolled out.
  • Which configuration these plans describe. Gitlab::ClickHouse.enabled_for_analytics? ignores its group argument and resolves to the instance-wide configured? && use_clickhouse_for_analytics?, so GitLab.com serves this field from ClickHouse and the PostgreSQL path is what self-managed instances without ClickHouse run. These plans were captured against GitLab.com's Postgres, roughly 3.7M workflow rows with a 3655-project namespace, which is a volume the PostgreSQL path does not normally serve. Treat the 30-day numbers as a fallback worst case rather than a steady-state cost.

ClickHouse: uniqExact over the key keeps exact state that grows with distinct keys in the window. It is bounded by the traversal-path prefix and the time window and gated by the default-off flag. uniqCombined is the approximate fallback if a very large namespace ever makes this expensive.

How to validate

Seed a group with a mix of internal and external sessions, then query each class. Internal sessions are rows with agent_type nil; external sessions have it set (for example claude-code, opencode) and may carry an agent_identity_id.

user = User.find_by_username('root')
Feature.enable(:ai_governance_dashboard)

query = <<~GQL
  query {
    group(fullPath: "your-group") {
      all:      aiGovernanceMetrics(agentClass: ALL)          { sessions { count } agents { count } }
      internal: aiGovernanceMetrics(agentClass: INTERNAL_DAP)  { sessions { count } agents { count } }
      external: aiGovernanceMetrics(agentClass: EXTERNAL)      { sessions { count } agents { count } }
    }
  }
GQL
puts GitlabSchema.execute(query, context: { current_user: user }).to_h

Validated locally against seeded data: 12 sessions in the current window from 8 distinct agent instances, on both backends:

agentClass sessions agents
ALL 12 8
INTERNAL_DAP 6 3
EXTERNAL 6 5

Identical results on the PostgreSQL and ClickHouse paths (toggle use_clickhouse_for_analytics), and 8 == 3 + 5 holds. The fixtures exercise every dimension of the key: two catalog-triggered web sessions collapse onto the same instance as an ambient session for the same user and project (the renamed-pair normalization), and two external sessions sharing user, project and agent_type but differing in agent_identity_id count as two machines.

Both service specs cover the machine dimension directly: two external sessions differing only in agent_identity_id count as two agents, and that example fails on the previous key.

References

EXTERNAL returns correct results over an empty set until real external sessions exist, which needs more than the producer MR above. !247477 (merged) is the receiving half. The producing half is the glab client, which has to register an identity, open a session on SessionStart and close it on SessionEnd. Until those hooks ship, a zero here is expected rather than a bug.

Two consequences worth knowing when the data does arrive. agent_identity_id is set by !247477 (merged)'s CreateService, which rejects a session whose identity does not resolve, so the machine dimension in the agents key works as soon as sessions flow. And if SessionStart ships ahead of SessionEnd, CleanStuckWorkflowsService will sweep external sessions older than 30 minutes to failed, because it has no environment predicate and external sessions write no checkpoints to refresh updated_at. That would make external agents read as almost entirely failing. Raised on !247477 (merged).

Edited by Andrew Jung

Merge request reports