Add rollout safeguards for session credits ingestion

Andrew Jungrequested to merge
ajung/29726-session-credits-rollout-safeguards into master

What does this MR do and why?

This MR is the corrective action for https://app.incident.io/gitlab/incidents/13873. Enabling the duo_workflow_session_credits_ingestion flag globally on 2026-09-04/05 made the cron cold-start against a 90-day backlog, pull up to 10,000 sessions per tick, fan out up to 10,000 child jobs, and run 50 CustomersDot calls at once, which saturated CustomersDot.

  • Cursor: the first cron run with no cursor yet seeds it at now minus 2 hours and skips the scan, so the old 90-day cold-start scan never runs again on a fresh instance. Every later run moves the cursor past every row it scans, whether the flag let that row dispatch or not, so a group or percentage rollout can't stall the pipeline the way it did before. The gap between deploy and flag-on isn't backfilled automatically; that's a deliberate console step (ClickHouse::SyncCursor.update_cursor_for(:duo_workflow_session_credits, <epoch seconds>)). GitLab.com already has a cursor, left where the Sept 5 rollback froze it, so once this deploys the cron walks forward from there at 1,000 rows a tick, dispatching nothing while the flag stays off. Setting it to two hours ago from a console before enabling the flag skips that catch-up: ClickHouse::SyncCursor.update_cursor_for(:duo_workflow_session_credits, 2.hours.ago.to_i).
  • Volume: MAX_ROWS_PER_RUN drops from 10,000 to 1,000, and the child job concurrency_limit drops from 50 to 5. Child jobs are one per root namespace per 100 ids in the batch, so worst case is 1,000 child jobs per tick with 5 running at a time.
  • Scheduling: the cron now gets an instance-specific minute offset via CronJobInitializer, based on the instance UUID, so every Sidekiq process runs it on the same schedule. Same pattern cron_for_service_ping already uses.
  • Back-off: a new SessionCredits::Backoff Redis latch. CustomersDot's kill switch latches for 6 hours the first time we see it. Connection errors, 5xx, and 429s latch for 30 minutes, but only once three separate root namespaces fail inside the same 15-minute window, so one heavy subscription can't pause everyone else. Each consecutive degradation latch escalates: 30 minutes, then 1, 2, and 4 hours, capped at MAX_BACKOFF_DURATION (4 hours). A successful fetch resets the escalation, and so does a latch-free hour after the last one expires, so the doubling only builds during a full outage; while some batches still get through, every back-off stays at 30 minutes. A new failure that lands while a latch is already running never shortens it. The cron skips whole cycles while latched and logs why. A successful fetch lifts only the kill-switch latch; a degradation latch expires on its own. Backoff.clear! is the console override.
  • Retries: already-enqueued child batches still retry with Sidekiq's default 25 attempts, but a failed batch now waits out the back-off before its next attempt instead of retrying within seconds, plus up to 30 minutes of jitter so the batches held during a latch do not all fire the moment it lifts. That floor applies to every error, not just CustomersDot ones, because any retry re-queries CustomersDot. A batch that starts while a degradation latch is already active raises for retry without calling CustomersDot at all, and its next attempt waits until the latch lifts or 30 minutes, whichever is longer. Later attempts keep Sidekiq's normal spacing, so the retry ladder spans about three weeks. Two caveats in the concurrency-limit middleware, both pre-existing: a retry that the limit defers loses Sidekiq's retry count, so under a backlog a permanently failing batch keeps retrying every 30 minutes until the flag is turned off instead of being dropped after 25 attempts; and a batch that was once resumed from the deferred buffer keeps its resume flag through retries, so its retries are not held to the 5-concurrent limit. Both need a follow-up on the middleware side.
  • Client: the client now recognizes CustomersDot's kill-switch error as reason: :disabled (no Sentry), and treats 5xx and 429 as reason: :server_error (no Sentry, triggers back-off).

Tested the first revision end to end against a local CustomersDot running the sibling MR: cold start, happy path, CustomersDot down, kill switch, and a real ClickHouse TIMEOUT_EXCEEDED. The threshold latch, retry spacing, always-advancing cursor, and 429 handling added in the second commit are covered by specs, not by a new end-to-end run, and so are the escalation, the latch-aware child worker, and the every-error retry floor added in the fourth commit.

References

How to set up and validate locally

  1. Truncate sync_cursors in ClickHouse and run Ai::DuoWorkflows::FetchSessionCreditsCronWorker.new.perform with the flag on or off. Expect the log line Seeded session credits cursor, no Postgres scan, and no child jobs.
  2. Move a billable workflow's updated_at to just past the seed time and rerun the cron: one child job runs.
  3. Enable the flag for a single group, create a billable workflow in a different group, and rerun: no child job runs, but the cursor still moves past it.
  4. Run Ai::DuoWorkflows::SessionCredits::Backoff.trigger!(reason: :server_error) and rerun the cron: expect the log line Skipping session credits cycle while backing off from CustomersDot.
  5. Stub the CustomersDot client to return a 500 and run Ai::DuoWorkflows::FetchNamespaceSessionCreditsWorker for three different root namespaces within the same 15-minute window: the third trips the latch (Backoff.reason returns :server_error). Running the same namespace three times does not trip it.
  6. Trigger a degradation back-off twice from a console with Ai::DuoWorkflows::SessionCredits::Backoff.trigger!(reason: :server_error), running Gitlab::Redis::SharedState.with { |r| r.del(Ai::DuoWorkflows::SessionCredits::Backoff::LATCH_KEY) } in between to clear the first latch. The second call's latch TTL should be about one hour, confirming the escalation.
  7. While that latch is active, run Ai::DuoWorkflows::FetchNamespaceSessionCreditsWorker.new.perform(<root namespace id>, [<workflow id>]) from a console. It should raise FetchError immediately, with no request going out to CustomersDot.

MR acceptance checklist

Edited by Andrew Jung

Merge request reports