## Summary Add a tenant-scoped GraphQL field `sessions(updatedAfter:)` on `gitlabCreditsUsage`. It returns every Duo Workflow session whose billed GitLab Credits changed since a given instant, with the session's lifetime credit total, per-model breakdown and a monotonic `updatedAt` that the caller persists as its next `updatedAfter`. Standard cursor pagination. This is the CustomersDot API half of the v2 session-credits sync ([gitlab-org/gitlab#617271](https://gitlab.com/gitlab-org/gitlab/-/work_items/617271)): GitLab keeps a per-subscription cursor and pulls only what changed. **Depends on #19201**, which builds `usage_billing_sessions`: one row per session with its total, per-model breakdown and an `UpdatedAt` written by `Billing::Usage::SyncSessionsCronJob` every 10 minutes. This issue only reads that table. ## Context <details> <summary>v1, its gaps, and who needs v2</summary> **v1.** `subscription(namespaceId: | licenseKey:).gitlabCreditsUsage(instanceId:, startDate:, endDate:).sessionCreditsUsed(workflowIds: [ID!]!)` (!16692, cache !16891, kill switch and query timeout !17077). A 15-minute GitLab cron (gitlab-org/gitlab!247620, safeguards !254266) scans `duo_workflows_workflows` for billable-status rows older than a 2-hour settle horizon, sends up to 100 ids per root namespace, and writes the totals to ClickHouse `duo_workflow_session_enrichments`. **Gaps**, from the [!247620 review](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/247620#note_3701109386): - Change detection is a proxy: GitLab infers "billed" from its own session status. A session that bills while paused and then fails is never fetched (#617273), and `BILLABLE_STATUSES` can drift from AI Gateway. - The 2-hour settle horizon is a guess, not a guarantee. - The cursor cannot cross a crowded second (#617272). - Each call is an unindexed scan of `usage_billing_enriched` with a JSON predicate on `Metadata`. Enabling it globally saturated CustomersDot ([INC-13873](https://app.incident.io/gitlab/incidents/13873); [production-engineering#29725](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29725), [#29726](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29726)). - `credit_finders_cache` never hits, because every tick sends a different id batch. **v2.** CustomersDot is the source of truth for "what changed": no status mirroring, no settle horizon, no id batches. Authorization stays as today: `namespaceId` (admin token) on GitLab.com, `licenseKey` + `instanceId` on self-managed. Choosing which namespaces to poll, per-tenant cursors, concurrency and back-off stay on the GitLab side. This replaces the server-side change log and admin-only "changed subscriptions" listing sketched in #617271. **Consumers.** - DAP Impact dashboard (&23225): private beta 2026-10-06 on v1; v2 makes a wider rollout safe. - AI Agent Artifacts `creditsUsed` column (#629047), blocked on the v1 rollout (#607369). - Closes production-engineering#29725 structurally. </details> ## Functional requirements 1. **Request.** ```graphql query { subscription(namespaceId: $namespaceId, licenseKey: $licenseKey) { gitlabCreditsUsage(instanceId: $instanceId, flowTypes: $flowTypes) { sessionsSyncedAt sessions(updatedAfter: $updatedAfter, first: $first, after: $after) { nodes { workflowId creditsUsed modelsUsed updatedAt creditsByModel { model creditsUsed } } pageInfo { hasNextPage endCursor } } } } } ``` 2. **`updatedAt`.** `usage_billing_sessions.UpdatedAt`: the time the #19201 job last wrote a changed result for the session. It moves forward only when `creditsUsed` or `creditsByModel` changes, so an unchanged session is not returned again. Any change counts, including a drop: a total that falls to zero (incident recovery only) is returned once with `0`, so the client's upsert corrects it. Serialized with microsecond precision through a new scalar, `ISO8601DateTimeMicro` (a subclass of `ISO8601DateTime` with `time_precision = 6`). In graphql-ruby the precision is one class-level setting shared by every `ISO8601DateTime` field, so it stays at 0 and existing fields keep their format. GitLab declares `$updatedAfter: ISO8601DateTimeMicro!`. 3. **`updatedAfter`.** Returns sessions with `updatedAt >= updatedAfter`. Inclusive: all rows of one job run share one `UpdatedAt`, and a client resuming from it re-reads that run instead of skipping rows that became visible later. Its upsert makes the repeat harmless. An `updatedAfter` in the future is a GraphQL error. 4. **No lost changes.** A session whose `updatedAt` becomes T is returned by every call with `updatedAfter <= T` made after T + δ, until its next change. - **δ = 1 minute.** The server never returns a session with `updatedAt > now() - δ`. The #19201 job is a single sequential writer (runs never overlap), so visibility follows `UpdatedAt` order; δ is only a guard against part-visibility lag between replicas. - **Freshness.** Credits reach the table 2–3 hours after usage (hourly consumption plus the 10-minute job). That lag is the consumption schedule, not a settle horizon: nothing is skipped because it came late. While the job is disabled, or skips runs because enrichment or consumption is behind, `updatedAt` stops advancing; the next run catches up from its stored watermark. `sessionsSyncedAt` (FR 11) exposes that watermark. 5. **`creditsUsed`.** `usage_billing_sessions.CreditsUsed`: billed, consumed, non-`SkipConsumption` credits, each event counted once, at its latest enriched version when its consumed marker was processed. Lifetime total: no month window and no lookback, every event of the session counts (#19201). - **Differs from v1 on purpose.** `sessionCreditsUsed` plain-sums enriched rows over the current month and can double-count re-enriched events. 6. **Event model.** As computed in #19201: each event's credits are attributed whole to `Metadata.llm_operations[0].model_id`. The event's other models are listed with `0`, and an event without `llm_operations` is counted under model `""`. 7. **`creditsByModel` and `modelsUsed`.** - **`creditsByModel`** is the `CreditsByModel` map as entries; they sum exactly to `creditsUsed`, and every model the session used is listed, with `0` when it received no credits. - **`modelsUsed`** lists the same models, ordered by `creditsUsed` descending, then by model. - **No effect on nodes.** `creditsByModel` is optional and doesn't change what a node is. A changed session returns its full breakdown, and clients replace it rather than merge. 8. **Ordering and paging.** Deterministic order by (`updatedAt`, `workflowId`, unique instance id). Default and maximum page size 500. Forward-only: `first`/`after` are supported, while `last`/`before` are a GraphQL error. Client loop: - page with `after = endCursor` while `hasNextPage`; - after each page, persist the highest `updatedAt` as the next `updatedAfter`. 9. **Lookback.** `updatedAfter` older than 99 days is a GraphQL error. Sessions last changed more than 99 days ago are unreachable through this field. 10. **v1 unchanged.** `sessionCreditsUsed(workflowIds:)` keeps working until GitLab migrates; deprecation is a follow-up. 11. **`sessionsSyncedAt`.** A nullable `ISO8601DateTimeMicro` field on `gitlabCreditsUsage`, next to `sessions`. It returns the #19201 job's watermark (`usage_billing_sync_states.watermark_at` for `sessions`): every consumed marker inserted before it is reflected in `sessions`. - Microsecond precision, as `updatedAt`. Not covered by the `sessions` kill switch: it reads no ClickHouse data. - `usage_billing_sync_states` is a Postgres table. The field is null until its `sessions` row exists. 12. **`flowTypes`.** The existing `gitlabCreditsUsage(flowTypes:)` argument applies to `sessions` as `FlowType IN (...)`, using the `FlowType` column of `usage_billing_sessions`. Without the argument, every flow type is returned. A client that uses a filter keeps a separate cursor per filter. ## Read path ```sql SELECT WorkflowId, UniqueInstanceId, RootNamespaceId, argMax(CreditsUsed, UpdatedAt) AS CreditsUsed, argMax(CreditsByModel, UpdatedAt) AS CreditsByModel, max(UpdatedAt) AS UpdatedAt FROM usage_billing_sessions WHERE ZuoraSubscriptionName = {subscription} [AND UniqueInstanceId IN ({resolve_unique_instance_ids(instance_id)})] -- licenseKey + instanceId [AND RootNamespaceId = {namespaceId}] -- namespaceId [AND FlowType IN ({flowTypes})] -- when `flowTypes` is given AND UpdatedAt >= {updatedAfter} GROUP BY WorkflowId, UniqueInstanceId, RootNamespaceId HAVING UpdatedAt <= now64(6) - INTERVAL 1 MINUTE AND (UpdatedAt, WorkflowId, UniqueInstanceId) > ({cursor}) -- when `after` is given ORDER BY UpdatedAt, WorkflowId, UniqueInstanceId LIMIT {first + 1} SETTINGS max_execution_time = …, timeout_overflow_mode = 'throw' ``` - **Cost.** `ZuoraSubscriptionName` is the sort-key prefix, so a call reads only the subscription's session rows (one or a few versions per session, narrow columns), never its events. - **`argMax`, not `FINAL`**, as elsewhere in usage billing (`WalletMismatchCheckerService`). - **`UpdatedAt >= updatedAfter` belongs in `WHERE`.** A session's newest version always has its highest `UpdatedAt`, so it passes whenever any older version does, and `argMax` still picks it. - **δ and the cursor belong in `HAVING`.** In `WHERE`, δ would drop a version written in the last minute and `argMax` would return the previous one: a stale total. In `HAVING`, the session waits out δ and then returns its latest version. The cursor compares against the aggregated `UpdatedAt`. - **Scaling option.** If a subscription's session count makes the range read slow, add `PROJECTION ORDER BY (ZuoraSubscriptionName, UpdatedAt)` in a follow-up. Not needed for the first version. - **Sort key.** #19201 changed the table to `ORDER BY (ZuoraSubscriptionName, UniqueInstanceId, RootNamespaceId, WorkflowId)` and added `FlowType` as a plain column, so the read groups by `RootNamespaceId` too. - **Scoping.** `namespaceId` resolves to the subscription as in v1 and adds `RootNamespaceId = {namespaceId}`, as v1 does. `licenseKey` + `instanceId` adds the `UniqueInstanceId` filter via `resolve_unique_instance_ids`, as in v1. - **A non-numeric namespace id is an error.** v1's `apply_root_namespace_id_filter` logs it and drops the filter. Here that would return every session of the subscription, across namespaces, so `sessions` raises instead. - **`FlowType` in `WHERE` is safe.** A session has one flow type, so filtering versions before `argMax` can't change which version wins. - **Cursor.** Opaque, encodes (`UpdatedAt` with microseconds, `WorkflowId`, `UniqueInstanceId`). - **Connection.** The existing `Gitlab::Graphql::Pagination::ClickHouseConnection` can't be reused: it adds the cursor in `WHERE` and needs Arel orders, but this read needs it in `HAVING`, over the aggregated `UpdatedAt`. `sessions` gets a small forward-only connection of its own. - **Complexity.** graphql-ruby scales a connection's complexity by its page size, so 500 nodes would exceed the schema limit of 250. The field sets a fixed `complexity`. The introspection limit (`INTROSPECTION_MAX_COMPLEXITY`) may also need a bump for the new types. ## Non-functional requirements - **Per-call cost is independent of the subscription's event volume.** It is proportional to the subscription's session count within the `ZuoraSubscriptionName` range. - **Query timeout.** Same mechanism as v1: `max_execution_time = SESSION_CREDITS_QUERY_TIMEOUT_SECONDS` with `timeout_overflow_mode = 'throw'`. - **Kill switch.** `usage_billing_disable_sessions_api` disables `sessions` only. v1's `sessionCreditsUsed` keeps its own switch, `usage_billing_disable_session_credits_api`, so either API can be turned off without the other. Both follow the `usage_billing_disable_<component>` convention: off by default, and an absent flag means enabled. The general `usage_billing_disable_api` disables both. Neither stops the #19201 job, which has its own switch (`usage_billing_disable_session_sync`). - `sessions` returns the error `sessions is temporarily disabled` with `extensions: { code: "FEATURE_DISABLED", field: "GitlabCreditsUsage.sessions" }`. The v2 client matches the code, not the text. - Implemented as a generic field option, `kill_switch_flag: :sessions_api` (`Extensions::KillSwitchExtension`), which v1 now uses too (`kill_switch_flag: :session_credits_api`); v1's message is unchanged. - **Reads only `usage_billing_sessions`.** No request reads `usage_billing_enriched` or `usage_billing_consumed`. - **Failure mode.** A timeout or disabled API returns a GraphQL error. The client keeps its cursor and retries on the next tick, so nothing is lost. CustomersDot adds no rate limit; GitLab bounds concurrency. ## Known limitations - **Re-enrichment and re-attribution** of already-consumed events aren't reflected until the session gets a new consumed marker (#19201, out of scope). Both happen only through incident recovery. - **One instance, several `UniqueInstanceId`s.** If an instance identifier resolves to several unique ids and one session spans them, the session comes back as one node per unique id with the same `workflowId`. An upsert by `workflowId` keeps one of them. Rare: it needs a session running across an instance re-registration. ## Acceptance criteria - [ ] `sessions` works for `namespaceId` and for `licenseKey` + `instanceId`; unauthenticated requests are rejected as today. - [ ] Self-managed requests only see sessions of the resolved `UniqueInstanceId`s. - [ ] Sessions with `updatedAt` newer than `now() - δ` are never returned, and a session with an unmerged older version returns only its latest one. - [ ] `creditsByModel` sums exactly to `creditsUsed`; `modelsUsed` lists exactly the models in `creditsByModel`, ordered by credits descending, then by model. - [ ] Pagination is deterministic by (`updatedAt`, `workflowId`, unique instance id); `endCursor` resumes without gaps or repeats; page size defaults to and is capped at 500; `last`/`before` are rejected. - [ ] `updatedAt` is serialized with microsecond precision. - [ ] Sessions that never had billed credits are absent; a session whose total drops to zero is returned once with `creditsUsed: 0`. - [ ] `updatedAfter` beyond 99 days or in the future returns a GraphQL error. - [ ] SaaS requests only see sessions of the requested `RootNamespaceId`; a non-numeric namespace id is an error. - [ ] `flowTypes` filters `sessions`; without it, every flow type is returned. - [ ] `usage_billing_disable_sessions_api` and the query timeout cover `sessions`; v1's kill switch doesn't. - [ ] `sessionsSyncedAt` returns the `sessions` sync watermark, and null before the sync state exists. - [ ] Per-call cost measured on staging for the largest subscription by session count; read path reviewed by @suraj_tripathy. - [ ] Request, type and model specs added (rows seeded directly into `usage_billing_sessions`); agent docs and `doc/process/session_credits_rollout.md` updated. ## References - Storage prerequisite: #19201 (`usage_billing_sessions` and `SyncSessionsCronJob`) - Review of the storage design: https://gitlab.com/gitlab-org/customers-gitlab-com/-/work_items/19049#note_3923694375 - v1 review and v2 draft: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/247620#note_3701109386 - Rollout and flag: https://gitlab.com/gitlab-org/gitlab/-/work_items/607369 - Epic: https://gitlab.com/groups/gitlab-org/-/work_items/23528 - Incident: INC-13873, production-engineering#29725, #29726 - CustomersDot v1: !16692, !16891, !17077; runbook `doc/process/session_credits_rollout.md` - Frontend consumer: https://gitlab.com/gitlab-org/gitlab/-/work_items/629047