Add v2 session credits table and CustomersDot query

Chandra Saripakarequested to merge
feat/629767-v2-credits-ingestion into master

What does this MR do and why?

This is step 1 of 3 of the v2 Duo Workflow session credits sync. It adds groundwork only: a new ClickHouse table and a CustomersDot client query. Nothing reads the table and nothing calls the query, so it is inert on merge. The feature flag duo_workflow_session_credits_sync_v2 ships with the worker MR together with its first caller, because rspec:feature-flags fails on a flag no code reads.

Why v2 exists: v1 infers which sessions billed from GitLab's own session status, so a session that bills and then moves to a failed status is never re-fetched. Its cursor stores whole epoch seconds with no tiebreak, so a second holding over 10,000 rows stalls it permanently. Enabling v1 globally saturated CustomersDot (incident INC-13873). v2 inverts it: CustomersDot reports which sessions changed since a timestamp and GitLab pages through them with a per-namespace cursor.

Credits are stored as Decimal64(4), not v1's Float64, to match the column CustomersDot computes them in. CustomersDot serialises creditsUsed as a GraphQL Float, so the acceptance criterion compares our stored totals against what the API returned at four decimal places.

Every unsuccessful response takes one path: the client classifies it (undefined field, 5xx, 429, disabled, unavailable) so the warning line names the reason, then raises ResponseError like the v1 query does. The kill switch for sessions is usage_billing_disable_sessions_api, recognised by extensions.code FEATURE_DISABLED. The query also returns sessionsSyncedAt, CustomersDot's sync watermark, so a caller can tell a stalled sync job from a quiet namespace. A null connection is reported as a failure rather than an empty page. pageInfo is returned as given: the caller stops paging when hasNextPage is false or endCursor is blank and resumes from its stored cursor on the next tick. The query carries the same 25 second read timeout as v1, because CustomersDot's ClickHouse limit equals GitLab's default HTTP read timeout.

Ingestion targets a separate table, duo_workflow_session_enrichments_v2, rather than the one v1 writes. That is deliberate and temporary: it exists so cold-start ingestion performance can be measured without disturbing v1 or the dashboards that read from it. Once the approach is proven, ingestion is repointed at the existing table.

Database review notes

duo_workflow_session_enrichments_v2 is a ReplacingMergeTree(updated_at) ordered by workflow_id, one row per Duo Workflow session, deduplicated on merge, so growth is bounded by the number of sessions rather than by time. It is write-only during the evaluation: the v2 ingest service in the follow-up MR writes it in gzipped CSV batches, and nothing reads it. It is dropped by a follow-up MR once ingestion is repointed at the existing table. updated_at has no default on purpose: it holds CustomersDot's clock, which trails ours, and a defaulted local time would win every merge.

References

Screenshots or screen recordings

Not applicable, this is a backend-only change with no user-visible surface.

Before After

How to set up and validate locally

  1. Run the ClickHouse migration: bundle exec rake gitlab:clickhouse:migrate
  2. Confirm the table exists and has the expected types, with a DESCRIBE on duo_workflow_session_enrichments_v2 — credits_used and the credits_by_model values should both be Decimal(18, 4), which is ClickHouse's canonical spelling of Decimal64(4).
  3. Run the specs: bundle exec rspec ee/spec/lib/gitlab/subscription_portal/subscription_usage_client_spec.rb -e "#get_sessions"
  4. Note there is nothing to exercise end to end yet: no worker calls the query, and CustomersDot has not shipped the sessions field. Until it does, a call logs CustomersDot sessions query failed: undefined_field and raises ResponseError; nothing calls it on merge, so there is no noise.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Chandra Saripaka

Merge request reports