Add the v2 session credits sync workers

Chandra Saripakarequested to merge
629767-v2-session-credits-workers into master

What does this MR do and why?

This MR implements steps 4 to 8 of #629767 on top of the merged v2 table and client from !258596 (merged).

v1 scans Postgres duo_workflows_workflows every 15 minutes and asks CustomersDot for credits per batch of workflow ids. That load caused INC-13873, where 50 concurrent uncached queries saturated CustomersDot.

v2 reverses the direction. For each root namespace, it asks CustomersDot for sessions changed since a cursor, pages through the result and writes it to ClickHouse. CustomersDot's sessions(updatedAfter:, first:, after:) API ships in customers-gitlab-com MR 17421. updatedAfter is inclusive and must be within 90 days. Pages hold up to 1000 sessions ordered by updatedAt.

  • IngestV2Service: maps session nodes to rows and inserts them as gzip CSV into duo_workflow_session_enrichments_v2. updated_at is CustomersDot's value, not the local clock. A node without a workflow id, or with an unparsable updatedAt, fails the whole page (invalid_rows), so nothing advances. A non-numeric creditsUsed also fails the page as invalid_rows. The cursor advance is computed from every node, including nil-credit ones. Nil credits are skipped and an explicit 0 is written.
  • V2Cursor: keeps the per-root-namespace position in ClickHouse sync_cursors as epoch microseconds. An unset cursor, or one that fell out of the window while the namespace was quiet, reads from a seed one hour inside the 90-day limit and the job logs cursor_reseeded: true. The opaque page token lives in Redis shared state with a 7-day TTL. The page token is keyed per cursor, carries the updatedAfter it was issued for, and survives cursor advances; a token whose window aged past the 90-day limit is dropped.
  • SyncNamespaceSessionCreditsWorker: loops get_sessions and Backoff.record_success, buffers up to 10 pages into one ClickHouse INSERT and stores the page token only after that INSERT, so a failure never skips rows, for up to 120 seconds. Its concurrency limit is 10 and it no longer defers on Postgres health. On GitLab.com a nil root namespace is a missing_root_namespace failure. It advances the cursor only when hasNextPage is false. ClickHouse and Redis errors, including the first cursor read, are logged failures (clickhouse_error, redis_error) and never raise for Sidekiq retry. A CustomersDot transport error is connection_error and counts toward the shared back-off, while ClickHouse connection errors are clickhouse_error and do not. hasNextPage with a blank endCursor is a malformed_response failure that writes and advances nothing. Every failure except the client's own ResponseError is reported to error tracking. Failures log stop_reason: error with a failure_reason and increment duo_workflow_session_credits_v2_fetch_failures_total{reason}. The next tick resumes.
  • SyncSessionCreditsCronWorker: skips the tick while Backoff is latched and logs namespace_jobs_buffered (the child jobs still buffered by the concurrency limit) at the start of each tick, which should be 0 when jobs finish between ticks. On GitLab.com it stops with siphon_disabled when Gitlab::ClickHouse.siphon_enabled?('duo_workflows_workflows') is false. Otherwise it finds root namespaces in ClickHouse siphon_duo_workflows_workflows (_siphon_watermark within 90 days), checks the flag per group (groups load with with_route, in slices of 1000) and deals the namespace jobs round-robin into one bucket per minute over the first 25 minutes of the tick. Postgres is not scanned. On self-managed it enqueues one job with a nil namespace when the instance flag is on.
  • Feature flag duo_workflow_session_credits_sync_v2 (gitlab_com_derisk, default off). The definition omits log_state_changes: true: Danger's flag parser rejects the key, and state changes are logged automatically while the 19.5 milestone is current. Re-add it if the rollout outlives the milestone.
  • Schedule: ai_duo_workflows_sync_session_credits_cron_worker runs every 30 minutes at a per-instance offset from the instance UUID, salted to differ from the v1 cron. all_queues.yml and sidekiq_queues.yml are regenerated.

No changelog, because all behaviour is behind a default-off flag. Not in this MR: writing to the real duo_workflow_session_enrichments table, dropping the _v2 table, removing v1 (#629771, !258986, !258987, !258988) and any UI.

References

Related to #629767

How to set up and validate locally

  1. In a Rails console, enable the flag for a group: Feature.enable(:duo_workflow_session_credits_sync_v2, group).
  2. Stub Gitlab::SubscriptionPortal::SubscriptionUsageClient#get_sessions to return one page of session nodes.
  3. Run Ai::DuoWorkflows::SyncSessionCreditsCronWorker.new.perform.
  4. Check the rows in duo_workflow_session_enrichments_v2 and the duo_workflow_session_credits_v2_{id} row in sync_cursors.
  5. Check the done line in log/sidekiq.log for root_namespace_id, pages_fetched, rows_written, calls_made, cursor_before, cursor_after, cursor_lag_s, cursor_reseeded, stop_reason, failure_reason, cdot_synced_at and cdot_sync_lag_s.

Verified locally: the current spec count is 139 examples across the seven spec files, run against ClickHouse and Redis. bin/rake gitlab:sidekiq:all_queues_yml:check gitlab:sidekiq:sidekiq_queues_yml:check passes and RuboCop is clean on every touched file. Not done yet: a GDK end-to-end run with the cron enabled and get_sessions stubbed. CI on master is the gate.

Decisions to confirm

@pshutsin, the issue left these open. Please confirm or object.

  • W2: seed one hour inside the 90-day window when the cursor is unset; a cursor that fell out of the window reseeds from the same point and is reported, instead of failing.
  • W3: page token in Redis shared state per cursor window, 7-day TTL.
  • W4: The cursor advances to the newest updatedAt written after each buffered INSERT, and the token keeps the pinned window across advances, so a long cold-start drain always progresses.
  • W6: no Sidekiq retries. A failed tick logs and the next tick resumes.
  • W7: counter duo_workflow_session_credits_v2_fetch_failures_total with a reason label. The v1 unlabeled counter stays until v1 is removed.
  • W9: discover namespaces from the siphon table in ClickHouse with a 90-day window, not Postgres.
  • W9 note: every active namespace is polled each tick. See the design note in the discussion for the alternative.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Chandra Saripaka

Merge request reports