Add product metrics for Duo tool governance

Rahul Barnwalrequested to merge
604497-hitl-guardrails-product-metrics into master

What does this MR do and why?

GitLab admins can set per-tool rules (Allow / Ask / Deny) for Duo agents, but today we have no product metrics that tell us if anyone uses this feature or how the approval prompts behave. This MR adds the measurement side only. No behavior changes.

It adds three Service Ping database metrics:

Metric What it counts
counts.count_distinct_namespaces_with_ai_tool_rules Namespaces that configured at least one tool rule
counts.count_distinct_namespaces_with_project_ai_tool_rules Namespaces that use project-level rule overrides
counts.count_abandoned_duo_workflow_tool_approvals Sessions stuck waiting for a tool approval for more than 24 hours (point-in-time snapshot)

The other side of the measurement (approval requested / resolved / denied-tool blocked) is instrumented as internal events emitted by the Duo Workflow Service in the companion MR below. Those event definitions live in the AI gateway repository and are measured warehouse-side from Snowplow, which is the established pattern for all gateway-emitted events (for example duo_workflow_tool_success). We do not define Service Ping metrics over them here on purpose: gateway events never pass through the Rails tracker, so both the instance Redis counters and the warehouse service-ping computation (which only reads events carrying the gitlab_service_ping context) would always report zero. A similar Rails-side shadow metric for gateway-measured behavior was removed for this reason in !153401 (merged).

Notes for reviewers:

  • The abandonment metric reads the workflow table directly instead of hooking CleanStuckWorkflowsService, because that service only looks at created/running workflows and never sees the approval-waiting state. A snapshot count is also naturally idempotent.
  • The 24 hour window is anchored to updated_at, so any write to the workflow row restarts it. The metric description documents this and the other caveats for data consumers.

References

Screenshots or screen recordings

No UI changes.

How to set up and validate locally

All three metrics were validated end to end on GDK, including through the settings UI. Steps:

  1. Make sure the governance settings page renders for your group. On GDK it needs SaaS simulation, the feature flag, and a paid plan with a Duo add-on on the group:

    # in <gdk-root>/env.runit, then gdk restart
export GITLAB_SIMULATE_SAAS=1
    # rails console
Feature.enable(:gitlab_duo_governance_settings, Group.find_by_full_path('gitlab-duo'))

    Note: env.runit only applies to GDK services. Start the console with the variable too, or the SaaS checks below return false:

    GITLAB_SIMULATE_SAAS=1 gdk rails console

    The seeded gitlab-duo group (from rake "gitlab:duo:setup[duo_enterprise]") already has the plan and add-on.

  2. Open http://127.0.0.1:3000/groups/gitlab-duo/-/settings/gitlab_duo/governance, go to the Tool management tab, and change any tool's Web access (for example set Create issue to Always ask). This creates an ai_tool_rules row.

  3. Read the metric values in rails console:

    # the helper recommended in doc/development/internal_analytics
require_relative 'spec/support/helpers/service_ping_helpers.rb'

ServicePingHelpers.get_current_usage_metric_value('counts.count_distinct_namespaces_with_ai_tool_rules')
ServicePingHelpers.get_current_usage_metric_value('counts.count_distinct_namespaces_with_project_ai_tool_rules')
ServicePingHelpers.get_current_usage_metric_value('counts.count_abandoned_duo_workflow_tool_approvals')

    Set a rule in a group that had none and the first metric goes up by 1. Add a project-scoped rule (a rule row with a project_id) and the second metric goes up by 1.

    If the settings page is not available for your test group (it needs a paid plan and a Duo add-on), create the rules from the console instead; the metrics count the rows either way:

    group = Group.find_by_full_path('your-test-group')
Ai::ToolRule.create!(namespace: group, tool_name: 'create_issue', web_access: 'ask', local_access: 'ask')
Ai::ToolRule.create!(namespace: group, project: group.projects.first, tool_name: 'create_merge_request', web_access: 'deny', local_access: 'deny')

  4. For the abandonment metric, put a session into the approval-waiting state: in Duo Agentic Chat, ask the agent to create an issue (with an Ask rule on the tool), and do not answer the approval prompt. Then backdate the workflow past the window:

    wf = Ai::DuoWorkflows::Workflow.with_status(:tool_call_approval_required).last
wf.update_column(:updated_at, 25.hours.ago)

    No working Duo chat? Create the row directly; the metric only reads the table:

    wf = Ai::DuoWorkflows::Workflow.new(user: User.first, project: Project.first, goal: 'test', workflow_definition: 'chat')
wf.save!(validate: false)
wf.update_columns(status: 8, updated_at: 25.hours.ago) # 8 = tool_call_approval_required

    The metric now counts it. At 1.hour.ago it does not. The boundary is strict: exactly 24 hours old is not counted.

Database review

Raw SQL generated by the three metrics:

SELECT COUNT(DISTINCT "ai_tool_rules"."namespace_id") FROM "ai_tool_rules";

SELECT COUNT(DISTINCT "ai_tool_rules"."namespace_id") FROM "ai_tool_rules"
WHERE "ai_tool_rules"."project_id" IS NOT NULL;

SELECT COUNT("duo_workflows_workflows"."id") FROM "duo_workflows_workflows"
WHERE "duo_workflows_workflows"."status" = 8
AND "duo_workflows_workflows"."updated_at" < (collection time - 24 hours);

Query plans (postgres.ai, gitlab-production-main):

  1. Namespaces with rules: Index Only Scan using idx_ai_tool_rules_ns_proj_tool_unique, 47.6 ms, ~4.9k rows scanned. Plan
  2. Namespaces with project rules: same index feeding a small in-memory sort, 33.2 ms, 75 rows. Plan
  3. Abandoned approvals: Index Only Scan using idx_workflows_status_updated_at_id, 1.22 s on a cold clone (almost all I/O read), ~59.8k matching rows. Plan
  • The first two are served by the existing unique index on (namespace_id, project_id, tool_name); ai_tool_rules is a small table (~4.9k rows).
  • The third is an exact match for the existing composite index idx_workflows_status_updated_at_id (status, updated_at, id) and never touches the table. The metric also runs through the batched counter, so production executes it as id-range batches on the same index rather than one statement.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Rahul Barnwal

Merge request reports