Add per-Duo-Workflow-session GitLab Credits ingestion and creditsUsed field

What

Adds creditsUsed to Duo Agent Platform session artifacts, so the AI Agent Artifacts dashboard has a per-session "credits used" column, plus the background ingestion that populates it.

Credits are computed in CustomersDot from ClickHouse billing events. GitLab cannot derive them, so this fetches them across the service boundary on a cron and stores them in the pre-existing duo_workflow_session_enrichments ClickHouse table. The GraphQL field then reads that table locally rather than putting a cross-service HTTP call in a dashboard page load.

This is the GitLab half (MR B) of issue 592156 (milestone 19.3, epic 22151). Zero migrations, Postgres and ClickHouse both.

Depends on CustomersDot

The flag must stay off until customers-gitlab-com!16692 is merged and deployed to CustomersDot production, which is roughly 3 hours after merge. Until then the sessionCreditsUsed field this calls does not exist in production. That cross-repo lead time is the reason for the feature flag: this MR can merge inert.

Changes

Bottom up:

1. Ai::DuoWorkflows::Workflow::BILLABLE_STATUSES and a with_billable_status scope. Deliberately not derived from TERMINAL_STATUSES, which includes failed (never bills) and omits the three pause states (which do bill).
2. SubscriptionUsageClient#get_session_credits(workflow_ids:), mirroring get_subscription_usage. Slices into pages of 100 ids (the CustomersDot cap) and merges the results.
3. SessionCredits::IngestService — resolves the date window, calls CustomersDot, writes to ClickHouse via insert_csv.
4. FetchNamespaceSessionCreditsWorker — one namespace batch per job. Transient CustomersDot failures raise a RetryError subclass so Sidekiq retries the batch; a batch that exhausts retries is logged at error level with its ids. FetchSessionCreditsCronWorker — cursor scan every 15 minutes, groups by root namespace, fans out.
5. creditsUsed on DuoWorkflowSessionArtifact, batch-loaded per page.

Feature flag duo_workflow_session_credits_ingestion, default off, gating the ingestion only. Rollout issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/607369

The GraphQL field is deliberately unflagged. It is nullable and returns null when no rows exist, which is the same thing it returns on any instance with ClickHouse analytics disabled.

How to test

The GraphQL field on its own

The field reads ClickHouse directly and has no dependency on the ingestion path, so it can be exercised without CustomersDot or the flag. Insert a row for a workflow you can see in the dashboard:

-- against the GDK ClickHouse, gitlab_clickhouse_development
INSERT INTO duo_workflow_session_enrichments (workflow_id, credits_used, updated_at)
VALUES (<WORKFLOW_ID>, 12.5, now64(6));

Then query it. Requires read_agent_artifacts on the group, and use_clickhouse_for_analytics enabled in application settings:

query {
  group(fullPath: "<GROUP_PATH>") {
    duoWorkflowSessionArtifacts(first: 10) {
      nodes {
        id
        creditsUsed
      }
    }
  }
}

Expect	Why it matters
creditsUsed: 12.5 on that session	the read path works
creditsUsed: null on a session with no row	absent must read null, never 0
insert a second row for the same id with a newer updated_at and a different value, then re-query	must return the newer value. The table is a ReplacingMergeTree, so unmerged duplicates are possible and the query uses argMax rather than a plain read
one ClickHouse query for a whole page of sessions	no N+1. Check the query log with several sessions in the page

The ingestion path end to end

Needs a reachable CustomersDot that has sessionCreditsUsed, so either a local CustomersDot with !16692 checked out, or staging.

# rails console
Feature.enable(:duo_workflow_session_credits_ingestion)

# A session only qualifies if its status is billable AND its last transition is
# older than the 2 hour settle horizon.
w = Ai::DuoWorkflows::Workflow.last
w.update!(status: 3, updated_at: 3.hours.ago)

Ai::DuoWorkflows::FetchSessionCreditsCronWorker.new.perform

Then confirm a row landed in duo_workflow_session_enrichments and that creditsUsed reflects it.

Worth checking the negative cases too, since they are where the behaviour is deliberate rather than incidental:

• Flag off, or use_clickhouse_for_analytics off: the cron is a complete no-op and the cursor does not move.
• A session whose last transition is inside the 2 hour horizon: not picked up yet.
• A failed session: never picked up, and reads null forever. Failed sessions never bill.
• A project-scoped session (namespace_id NULL, project_id set): still dispatched, under its root namespace. See the first gap below.

Notes for reviewers

Four things that are load-bearing and not obvious from the diff.

The billable status set is not the terminal status set. AI Gateway emits a billing event at every billable transition including the three pause states, so a session's total keeps growing over its life and must be re-fetched on later transitions. TERMINAL_STATUSES is a different set and reusing it would both over-fetch (failed) and under-fetch (the pause states).

Sessions are scoped to either a namespace or a project, never both (CHECK (num_nonnulls(namespace_id, project_id) = 1)), and a non-NULL namespace_id may be a subgroup while CustomersDot subscriptions key on the root. The cron therefore selects project_id as well and resolves roots in bulk through namespaces.traversal_ids, two queries per run rather than root_ancestor per row. Resolving only namespace_id would have silently dropped every project-scoped session.

The cursor bound is > cursor_time, not >= cursor_time + 1.second. ClickHouse::SyncCursor stores a UInt64, so the high-water mark is a whole second. If MAX_ROWS_PER_RUN truncates mid-second, rows later in that same second were never dispatched, and skipping forward past them loses them permanently. Rounding toward re-processing is safe because the target table is a ReplacingMergeTree keyed on workflow_id, so a re-fetch is an upsert. There is a regression spec that stubs the row cap and asserts the truncated row is rescanned.

urgency :throttled on the child worker is intentional, not a downgrade. concurrency_limit -> { 50 } deliberately buffers the fan-out, and buffered time is added to the measured queueing duration (Gitlab::InstrumentationHelper.queue_duration_for_job defaults to with_buffering_duration: true). Under :low that self-inflicted delay would be scored against a 60 second queueing target and breach it by design. :throttled has no queueing target and the same 300 second execution target. It also declares worker_has_external_dependencies!, which raises the error alerting threshold since a failure here is most likely CustomersDot rather than us.

Both workers route to the default queue and land on the catchall shard. Note tags :clickhouse is informational and does not affect routing.

Expected load

Per 15-minute cycle:

• One Postgres scan, keyset-bounded, capped at 10k rows, on the (status, updated_at, id) index, plus two bulk plucks for root namespace resolution.
• Job count: under 100 in the best case (sessions cluster into a few root namespaces), up to 10k in the pathological case (every session a different root namespace). Fan-out drains at concurrency_limit 50, so at most 50 jobs run at once.
• Exactly one CustomersDot GraphQL call per namespace job (batches are capped at 100 ids by construction).

No production baseline yet for how many sessions transition into billable statuses per 15 minutes on GitLab.com; the feature flag lets us roll out on GitLab.com first, watch the CustomersDot-side ClickHouse metrics, and tune from there.

Transient failures retry through Sidekiq's normal ladder (25 retries with exponential backoff); every failed attempt is counted and logged by the ingest service, and a batch that exhausts retries is logged at error level with its workflow ids so it can be replayed.

Test coverage

625 examples, 0 failures across the seven affected spec files, verified locally at this commit. bin/rubocop clean. Generated artifacts regenerated rather than hand-edited: all_queues.yml, config/sidekiq_queues.yml, doc/api/graphql/reference/_index.md, public/-/graphql/introspection_result.json.

Round 3 (worker retries + client pagination): 275 examples, 0 failures across the four affected spec files at the current head.

The specs that matter most are the negative ones: absent session reads null rather than zero, unmerged duplicates resolve to the newest row, the cursor does not advance when nothing was dispatched, and a truncated mid-second row is rescanned.

Follow-ups, not in this MR

• Reduce the child job count. The fan-out is one job per namespace, so 10,000 billable sessions across 10,000 distinct root namespaces is 10,000 jobs. Packing several namespaces per job would cut that by orders of magnitude with no change to CustomersDot pressure, since concurrency_limit bounds concurrent calls either way. Worth measuring the real distinct-namespace rate first.
• Pass the date window down from the cron. IngestService re-derives min(created_at) per job, which the cron could compute once from rows it has already scanned.
• A cross-subscription batch endpoint on CustomersDot is the only thing that reduces actual HTTP volume, rather than just job count.
• Sessions older than 90 days have their window clamped and may miss their earliest events. Accepted.
• Self-managed instances without ClickHouse analytics have no credits source and read null.

Database review

All writes in this MR go to ClickHouse. The only Postgres touches are reads. Both workers declare defer_on_database_health_signal :gitlab_main_org, [:duo_workflows_workflows].

1. Cron scan, steady state FetchSessionCreditsCronWorker#scan (with_billable_status scope, updated_at window, ORDER BY updated_at, id, LIMIT 10000). Rides idx_workflows_status_updated_at_id (status, updated_at, id): 12ms, 27 buffers on a 15-minute window. Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/54490/commands/157267

2. Same scan, cold start / worst case 90-day window, the floor at MAX_WINDOW_DAYS so this is the widest window the scan can ever hit. Parallel index scan over ~611k rows per worker (~1.8M total) with a top-N heapsort, ~35s. The 5-value status IN-list rules out a single ordered index walk, which is why this isn't as cheap as query 1. Bounded by: LIMIT 10000 per tick, cursor advances each tick so the window shrinks as backlog drains, cron runs at urgency :low with retries disabled, and this shape only occurs once per cold start. Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/54490/commands/157268

3. root_namespace_ids_for pluck 1 SELECT id, namespace_id FROM projects WHERE id IN (up to 10k literal ids), a canonical PK IN-list. The linked plan's cost is dominated by a CTE that samples ids for testing, the app never runs it. App-relevant node is the inner per-id index scan, ~4ms per row cold-cache. Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/54490/commands/157270

4. root_namespace_ids_for pluck 2 SELECT id, traversal_ids[1] FROM namespaces WHERE id IN (...). traversal_ids[1] is computed per row after a namespaces_pkey lookup, never used as a predicate, so no index on traversal_ids is involved. Same CTE-sampling caveat as query 3; app-relevant node is the pkey index scan, ~3.4ms per row cold-cache. Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/54490/commands/157271

5. IngestService#start_date SELECT MIN(created_at) FROM duo_workflows_workflows WHERE id IN (batch of at most 100 ids). Same CTE-sampling caveat; app-relevant node is 100 pkey lookups plus the aggregate. Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/54490/commands/157272

Note on introspection

The experiment: { milestone: '19.3' } marker on creditsUsed intentionally shows up as isDeprecated: true with a Status: Experiment reason in the introspection dump. That is GitLab's standard encoding for experiment fields, not an accident.