Add sessions(updatedAfter:): per-session GitLab Credits changes with cursor pagination

Summary

Add a tenant-scoped GraphQL field sessions(updatedAfter:) on gitlabCreditsUsage. It returns every Duo Workflow session whose billed GitLab Credits changed since a given instant, with the session's lifetime credit total, per-model breakdown and a monotonic updatedAt that the caller persists as its next updatedAfter. Standard cursor pagination.

This is the CustomersDot API half of the v2 session-credits sync (gitlab-org/gitlab#617271): GitLab keeps a per-subscription cursor and pulls only what changed.

Depends on #19201, which builds usage_billing_sessions: one row per session with its total, per-model breakdown and an UpdatedAt written by Billing::Usage::SyncSessionsCronJob every 10 minutes. This issue only reads that table.

Context

v1, its gaps, and who needs v2

v1. subscription(namespaceId: | licenseKey:).gitlabCreditsUsage(instanceId:, startDate:, endDate:).sessionCreditsUsed(workflowIds: [ID!]!) (!16692, cache !16891, kill switch and query timeout !17077). A 15-minute GitLab cron (gitlab!247620 (merged), safeguards !254266) scans duo_workflows_workflows for billable-status rows older than a 2-hour settle horizon, sends up to 100 ids per root namespace, and writes the totals to ClickHouse duo_workflow_session_enrichments.

Gaps, from the !247620 review:

  • Change detection is a proxy: GitLab infers "billed" from its own session status. A session that bills while paused and then fails is never fetched (#617273), and BILLABLE_STATUSES can drift from AI Gateway.
  • The 2-hour settle horizon is a guess, not a guarantee.
  • The cursor cannot cross a crowded second (#617272).
  • Each call is an unindexed scan of usage_billing_enriched with a JSON predicate on Metadata. Enabling it globally saturated CustomersDot (INC-13873; production-engineering#29725, #29726).
  • credit_finders_cache never hits, because every tick sends a different id batch.

v2. CustomersDot is the source of truth for "what changed": no status mirroring, no settle horizon, no id batches. Authorization stays as today: namespaceId (admin token) on GitLab.com, licenseKey + instanceId on self-managed. Choosing which namespaces to poll, per-tenant cursors, concurrency and back-off stay on the GitLab side. This replaces the server-side change log and admin-only "changed subscriptions" listing sketched in #617271.

Consumers.

  • DAP Impact dashboard (&23225): private beta 2026-10-06 on v1; v2 makes a wider rollout safe.
  • AI Agent Artifacts creditsUsed column (#629047), blocked on the v1 rollout (#607369).
  • Closes production-engineering#29725 structurally.

Functional requirements

  1. Request.

    query {
  subscription(namespaceId: $namespaceId, licenseKey: $licenseKey) {
    gitlabCreditsUsage(instanceId: $instanceId, flowTypes: $flowTypes) {
      sessionsSyncedAt
      sessions(updatedAfter: $updatedAfter, first: $first, after: $after) {
        nodes {
          workflowId creditsUsed modelsUsed updatedAt
          creditsByModel { model creditsUsed }
        }
        pageInfo { hasNextPage endCursor }
      }
    }
  }
}

  2. updatedAt. usage_billing_sessions.UpdatedAt: the time the #19201 job last wrote a changed result for the session. It moves forward only when creditsUsed or creditsByModel changes, so an unchanged session is not returned again. Any change counts, including a drop: a total that falls to zero (incident recovery only) is returned once with 0, so the client's upsert corrects it. Serialized with microsecond precision through a new scalar, ISO8601DateTimeMicro (a subclass of ISO8601DateTime with time_precision = 6). In graphql-ruby the precision is one class-level setting shared by every ISO8601DateTime field, so it stays at 0 and existing fields keep their format. GitLab declares $updatedAfter: ISO8601DateTimeMicro!.

  3. updatedAfter. Returns sessions with updatedAt >= updatedAfter. Inclusive: all rows of one job run share one UpdatedAt, and a client resuming from it re-reads that run instead of skipping rows that became visible later. Its upsert makes the repeat harmless. An updatedAfter in the future is a GraphQL error.

  4. No lost changes. A session whose updatedAt becomes T is returned by every call with updatedAfter <= T made after T + δ, until its next change.

    • δ = 1 minute. The server never returns a session with updatedAt > now() - δ. The #19201 job is a single sequential writer (runs never overlap), so visibility follows UpdatedAt order; δ is only a guard against part-visibility lag between replicas.
    • Freshness. Credits reach the table 2–3 hours after usage (hourly consumption plus the 10-minute job). That lag is the consumption schedule, not a settle horizon: nothing is skipped because it came late. While the job is disabled, or skips runs because enrichment or consumption is behind, updatedAt stops advancing; the next run catches up from its stored watermark. sessionsSyncedAt (FR 11) exposes that watermark.

  5. creditsUsed. usage_billing_sessions.CreditsUsed: billed, consumed, non-SkipConsumption credits, each event counted once, at its latest enriched version when its consumed marker was processed. Lifetime total: no month window and no lookback, every event of the session counts (#19201).

    • Differs from v1 on purpose. sessionCreditsUsed plain-sums enriched rows over the current month and can double-count re-enriched events.

  6. Event model. As computed in #19201: each event's credits are attributed whole to Metadata.llm_operations[0].model_id. The event's other models are listed with 0, and an event without llm_operations is counted under model "".

  7. creditsByModel and modelsUsed.

    • creditsByModel is the CreditsByModel map as entries; they sum exactly to creditsUsed, and every model the session used is listed, with 0 when it received no credits.
    • modelsUsed lists the same models, ordered by creditsUsed descending, then by model.
    • No effect on nodes. creditsByModel is optional and doesn't change what a node is. A changed session returns its full breakdown, and clients replace it rather than merge.

  8. Ordering and paging. Deterministic order by (updatedAt, workflowId, unique instance id). Default and maximum page size 500. Forward-only: first/after are supported, while last/before are a GraphQL error. Client loop:

    • page with after = endCursor while hasNextPage;
    • after each page, persist the highest updatedAt as the next updatedAfter.

  9. Lookback. updatedAfter older than 99 days is a GraphQL error. Sessions last changed more than 99 days ago are unreachable through this field.

  10. v1 unchanged. sessionCreditsUsed(workflowIds:) keeps working until GitLab migrates; deprecation is a follow-up.

  11. sessionsSyncedAt. A nullable ISO8601DateTimeMicro field on gitlabCreditsUsage, next to sessions. It returns the #19201 job's watermark (usage_billing_sync_states.watermark_at for sessions): every consumed marker inserted before it is reflected in sessions.

    • Microsecond precision, as updatedAt. Not covered by the sessions kill switch: it reads no ClickHouse data.
    • usage_billing_sync_states is a Postgres table. The field is null until its sessions row exists.

  12. flowTypes. The existing gitlabCreditsUsage(flowTypes:) argument applies to sessions as FlowType IN (...), using the FlowType column of usage_billing_sessions. Without the argument, every flow type is returned. A client that uses a filter keeps a separate cursor per filter.

Read path

SELECT
  WorkflowId, UniqueInstanceId, RootNamespaceId,
  argMax(CreditsUsed, UpdatedAt)    AS CreditsUsed,
  argMax(CreditsByModel, UpdatedAt) AS CreditsByModel,
  max(UpdatedAt)                    AS UpdatedAt
FROM usage_billing_sessions
WHERE ZuoraSubscriptionName = {subscription}
  [AND UniqueInstanceId IN ({resolve_unique_instance_ids(instance_id)})]  -- licenseKey + instanceId
  [AND RootNamespaceId = {namespaceId}]                                   -- namespaceId
  [AND FlowType IN ({flowTypes})]                                         -- when `flowTypes` is given
  AND UpdatedAt >= {updatedAfter}
GROUP BY WorkflowId, UniqueInstanceId, RootNamespaceId
HAVING UpdatedAt <= now64(6) - INTERVAL 1 MINUTE
   AND (UpdatedAt, WorkflowId, UniqueInstanceId) > ({cursor})    -- when `after` is given
ORDER BY UpdatedAt, WorkflowId, UniqueInstanceId
LIMIT {first + 1}
SETTINGS max_execution_time = …, timeout_overflow_mode = 'throw'
  • Cost. ZuoraSubscriptionName is the sort-key prefix, so a call reads only the subscription's session rows (one or a few versions per session, narrow columns), never its events.
  • argMax, not FINAL, as elsewhere in usage billing (WalletMismatchCheckerService).
  • UpdatedAt >= updatedAfter belongs in WHERE. A session's newest version always has its highest UpdatedAt, so it passes whenever any older version does, and argMax still picks it.
  • δ and the cursor belong in HAVING. In WHERE, δ would drop a version written in the last minute and argMax would return the previous one: a stale total. In HAVING, the session waits out δ and then returns its latest version. The cursor compares against the aggregated UpdatedAt.
  • Scaling option. If a subscription's session count makes the range read slow, add PROJECTION ORDER BY (ZuoraSubscriptionName, UpdatedAt) in a follow-up. Not needed for the first version.
  • Sort key. #19201 changed the table to ORDER BY (ZuoraSubscriptionName, UniqueInstanceId, RootNamespaceId, WorkflowId) and added FlowType as a plain column, so the read groups by RootNamespaceId too.
  • Scoping. namespaceId resolves to the subscription as in v1 and adds RootNamespaceId = {namespaceId}, as v1 does. licenseKey + instanceId adds the UniqueInstanceId filter via resolve_unique_instance_ids, as in v1.
    • A non-numeric namespace id is an error. v1's apply_root_namespace_id_filter logs it and drops the filter. Here that would return every session of the subscription, across namespaces, so sessions raises instead.
  • FlowType in WHERE is safe. A session has one flow type, so filtering versions before argMax can't change which version wins.
  • Cursor. Opaque, encodes (UpdatedAt with microseconds, WorkflowId, UniqueInstanceId).
  • Connection. The existing Gitlab::Graphql::Pagination::ClickHouseConnection can't be reused: it adds the cursor in WHERE and needs Arel orders, but this read needs it in HAVING, over the aggregated UpdatedAt. sessions gets a small forward-only connection of its own.
  • Complexity. graphql-ruby scales a connection's complexity by its page size, so 500 nodes would exceed the schema limit of 250. The field sets a fixed complexity. The introspection limit (INTROSPECTION_MAX_COMPLEXITY) may also need a bump for the new types.

Non-functional requirements

  • Per-call cost is independent of the subscription's event volume. It is proportional to the subscription's session count within the ZuoraSubscriptionName range.
  • Query timeout. Same mechanism as v1: max_execution_time = SESSION_CREDITS_QUERY_TIMEOUT_SECONDS with timeout_overflow_mode = 'throw'.
  • Kill switch. usage_billing_disable_sessions_api disables sessions only. v1's sessionCreditsUsed keeps its own switch, usage_billing_disable_session_credits_api, so either API can be turned off without the other. Both follow the usage_billing_disable_<component> convention: off by default, and an absent flag means enabled. The general usage_billing_disable_api disables both. Neither stops the #19201 job, which has its own switch (usage_billing_disable_session_sync).
    • sessions returns the error sessions is temporarily disabled with extensions: { code: "FEATURE_DISABLED", field: "GitlabCreditsUsage.sessions" }. The v2 client matches the code, not the text.
    • Implemented as a generic field option, kill_switch_flag: :sessions_api (Extensions::KillSwitchExtension), which v1 now uses too (kill_switch_flag: :session_credits_api); v1's message is unchanged.
  • Reads only usage_billing_sessions. No request reads usage_billing_enriched or usage_billing_consumed.
  • Failure mode. A timeout or disabled API returns a GraphQL error. The client keeps its cursor and retries on the next tick, so nothing is lost. CustomersDot adds no rate limit; GitLab bounds concurrency.

Known limitations

  • Re-enrichment and re-attribution of already-consumed events aren't reflected until the session gets a new consumed marker (#19201, out of scope). Both happen only through incident recovery.
  • One instance, several UniqueInstanceIds. If an instance identifier resolves to several unique ids and one session spans them, the session comes back as one node per unique id with the same workflowId. An upsert by workflowId keeps one of them. Rare: it needs a session running across an instance re-registration.

Acceptance criteria

  • sessions works for namespaceId and for licenseKey + instanceId; unauthenticated requests are rejected as today.
  • Self-managed requests only see sessions of the resolved UniqueInstanceIds.
  • Sessions with updatedAt newer than now() - δ are never returned, and a session with an unmerged older version returns only its latest one.
  • creditsByModel sums exactly to creditsUsed; modelsUsed lists exactly the models in creditsByModel, ordered by credits descending, then by model.
  • Pagination is deterministic by (updatedAt, workflowId, unique instance id); endCursor resumes without gaps or repeats; page size defaults to and is capped at 500; last/before are rejected.
  • updatedAt is serialized with microsecond precision.
  • Sessions that never had billed credits are absent; a session whose total drops to zero is returned once with creditsUsed: 0.
  • updatedAfter beyond 99 days or in the future returns a GraphQL error.
  • SaaS requests only see sessions of the requested RootNamespaceId; a non-numeric namespace id is an error.
  • flowTypes filters sessions; without it, every flow type is returned.
  • usage_billing_disable_sessions_api and the query timeout cover sessions; v1's kill switch doesn't.
  • sessionsSyncedAt returns the sessions sync watermark, and null before the sync state exists.
  • Per-call cost measured on staging for the largest subscription by session count; read path reviewed by @suraj_tripathy.
  • Request, type and model specs added (rows seeded directly into usage_billing_sessions); agent docs and doc/process/session_credits_rollout.md updated.

References

Edited by Pavel Shutsin