Schedule and paginate Claude Code session list ingestion

Hitesh Raghuvanshirequested to merge
623177-anthropic-session-list-sync into master

What does this MR do and why?

Pulls Claude Code sessions from the Claude Compliance API (Anthropic) every 6 hours and records them as AI governance sessions. Behind the claude_code_compliance_api_ingestion feature flag (wip, default off, checked per top-level group).

  • ScheduleIngestionWorker (cron, every 6 hours) starts one session-list walk per enabled integration. Jobs go out in waves of 10, at most 5 minutes apart, so they don't all hit Sidekiq at once (same fix as !262205 (merged)). Each job keeps its namespace in the logging context.
  • SyncSessionsWorker reads one page of up to 500 sessions updated in the last 12 hours, then enqueues itself for the next page. For each claude_code session it finds the GitLab user by email (UserResolver) and records it with the existing Sessions::CreateService. Sessions it can't attribute are dropped and logged.
  • When a walk reaches the last page, it sets last_successful_sync_at. A failed walk is not retried (retry: 0); the next run lists the same window again.
  • Worker settings: concurrency_limit 10 and urgency :throttled (as in !262205 (merged)).
  • UserResolver now logs a missing Anthropic email as no_anthropic_email.

Database queries

ai_compliance_anthropic_integrations is empty on GitLab.com, so plans were run after inserting 2,000 integrations on the Database Lab clone. All use an index and return rows.

Query Where Plan
Batch start, boundary, rows (enabled scope) schedule_ingestion_worker.rb:33, integration.rb:29 start, boundary, rows
Namespace and route preload integration.rb:30 namespaces, routes
Load integration, set last_successful_sync_at sync_sessions_worker.rb:32, :46 select, update
SQL 
SELECT "ai_compliance_anthropic_integrations"."id" FROM "ai_compliance_anthropic_integrations" WHERE "ai_compliance_anthropic_integrations"."enabled" = TRUE ORDER BY "ai_compliance_anthropic_integrations"."id" ASC LIMIT 1;
SELECT "ai_compliance_anthropic_integrations"."id" FROM "ai_compliance_anthropic_integrations" WHERE "ai_compliance_anthropic_integrations"."enabled" = TRUE AND ("ai_compliance_anthropic_integrations"."id" >= 1) ORDER BY "ai_compliance_anthropic_integrations"."id" ASC LIMIT 1 OFFSET 100;
SELECT "ai_compliance_anthropic_integrations".* FROM "ai_compliance_anthropic_integrations" WHERE "ai_compliance_anthropic_integrations"."enabled" = TRUE AND ("ai_compliance_anthropic_integrations"."id" >= 1) AND ("ai_compliance_anthropic_integrations"."id" < 143);
SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."id" IN (6543, 9970, ...);
SELECT "routes".* FROM "routes" WHERE "routes"."source_type" = 'Namespace' AND "routes"."source_id" IN (6543, 9970, ...);
SELECT "ai_compliance_anthropic_integrations".* FROM "ai_compliance_anthropic_integrations" WHERE "ai_compliance_anthropic_integrations"."id" = 500 LIMIT 1;
UPDATE "ai_compliance_anthropic_integrations" SET "updated_at" = '2026-10-22 12:00:00', "last_successful_sync_at" = '2026-10-22 12:00:00' WHERE "ai_compliance_anthropic_integrations"."id" = 500;

Out of scope

How to set up and validate locally

bundle exec rspec ee/spec/workers/ai/compliance/anthropic ee/spec/lib/ai/compliance/anthropic/user_resolver_spec.rb ee/spec/models/ai/compliance/anthropic/integration_spec.rb

On GDK, with an enabled integration that has a real Compliance Access Key:

  1. Feature.enable(:claude_code_compliance_api_ingestion, group)
  2. Ai::Compliance::Anthropic::SyncSessionsWorker.new.perform(integration.id)
  3. Check the new Ai::Governance::Session rows and the integration's last_successful_sync_at.

References

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Hitesh Raghuvanshi

Merge request reports