Stagger compliance framework evaluation scheduling

What does this MR do and why?

Every 12 hours, ComplianceManagement::FrameworkEvaluationSchedulerWorker enqueues one ProjectComplianceEvaluatorWorker job per batch of 100 projects in each active framework, all at once. On GitLab.com that's about 3,930 database-heavy jobs per run. The burst saturated the shared Sidekiq PgBouncer pool and contributed to recent incidents.

This MR:

• Staggers the scheduler (behind stagger_compliance_framework_evaluation_scheduling). Jobs are scheduled in waves of 30 with bulk_perform_in, spread across a 10-hour window that leaves a 2-hour buffer before the next run. The gap between waves is capped at 5 minutes so small instances finish quickly. On GitLab.com this gives about 131 waves, roughly 4.5 minutes apart.
• Caps the evaluator at 30 concurrent jobs (not flagged). The wave size matches the cap, so each wave gets free slots. The worker moves from urgency :low to :throttled, because jobs waiting in the concurrency limiter queue would count against :low's 60s queueing target.
• Sorts project IDs in framework_project_mappings, so batches stay the same between runs and until_executed deduplication can match repeated batches.
• Logs re-schedules. When a job hits the runtime limit and re-schedules its remaining projects, it records rescheduled_project_count in its done log.

Things to know

• Turning the flag off does not fully revert this MR. It restores the old all-at-once scheduling, but the cap of 30 and :throttled urgency stay. With the flag off, each run buffers about 3,900 jobs in the concurrency limiter queue, which drain in roughly 8–9.5 hours. Enable the flag soon after deploy.
• Some user-triggered evaluations can be delayed until the next scheduled wave (up to about 10 hours). Scheduled jobs hold their deduplication key until they run. A job with identical arguments is dropped in favor of the scheduled one. This affects re-evaluation after requirement or control changes (ProjectsComplianceEnqueueWorker, which builds the same ascending 100-ID batches) and assigning a framework that has only one project. The evaluation still happens, just later. This delay is accepted for now.

Production data behind the numbers

• 3,930 scheduler jobs per run across 1,629 active frameworks (production clone via Joe).
• Kibana, Oct 10–14: scheduler-triggered jobs averaged 95–153s per cycle (about 125s overall), with many at 200–290s. About 2,800 more jobs per cycle come from runtime-limiter re-schedules.
• At 30 concurrent jobs, one cycle clears in about 8–9.5 hours, about 78% use of the window at 125s.
• 30 concurrent jobs use about 30 of the 69 Sidekiq PgBouncer connections.

References

• #633677
• Rollout issue: #633822
• Earlier concurrency limit MR: !261408 (closed)
• gitlab-com/gl-infra/tenant-scale/tenant-services/team#517 (closed)

Screenshots or screen recordings

N/A, backend worker change with no UI.

How to set up and validate locally

1. Make sure more than 30 frameworks have a requirement with a control and at least one assigned project, so the scheduler creates more than one wave.
2. In the Rails console, enable the flag and run the scheduler:

   Feature.enable(:stagger_compliance_framework_evaluation_scheduling)
   ComplianceManagement::FrameworkEvaluationSchedulerWorker.new.perform

3. Check the scheduled times:

   Gitlab::SidekiqSharding::Validator.allow_unrouted_sidekiq_calls do
     Sidekiq::ScheduledSet.new
       .select { |job| job.klass == 'ComplianceManagement::ProjectComplianceEvaluatorWorker' }
       .map { |job| job.at.to_i }.tally
   end

   Jobs are grouped 30 per timestamp, each group up to 5 minutes after the previous one. Locally, with 36 jobs, I saw 30 at +1s and 6 at +301s, and all ran successfully.
4. Run the scheduler again while the second wave is still waiting. The waiting jobs are not duplicated.
5. Disable the flag and run the scheduler again. Jobs are enqueued immediately, as before.
6. ComplianceManagement::ProjectComplianceEvaluatorWorker.get_concurrency_limit returns 30.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.