Run VSA incremental aggregation one organization per job

What does this MR do and why?

On Cells, every Value Stream Analytics aggregation job must work for one organization only. Today Analytics::CycleAnalytics::IncrementalWorker processes the aggregations of every organization in one cron job, with no organization context. A large organization can use up the 200-second runtime budget of the others.

This is the first of four MRs, one per VSA aggregation cron worker. It covers the incremental worker.

  • Behind the vsa_aggregation_per_organization flag (gitlab_com_derisk, default off), IncrementalWorker schedules one OrganizationIncrementalWorker job per organization, with organization: in the job context. Organizations without aggregations finish after one cheap query (see Database). With the flag off, IncrementalWorker runs the same loop inline for the whole instance, as it does today.
  • OrganizationIncrementalWorker loads only the aggregations of its organization (Aggregation.load_batch(..., organization:)), with its own runtime limit.
  • OrganizationIncrementalWorker is a separate class because CronjobQueue.context_for_arguments replaces the batch context of a cron worker's own jobs with caller_id: "Cronjob", which would drop the organization.
  • Removes the Scalability/CronWorkerContext disable from IncrementalWorker.
  • The new worker uses data_consistency :sticky, like StageAggregationWorker, and retry: false, like the cron worker it replaces, because the next cron run picks up any leftover work.
  • Parentable gets a for_organization scope. StageAggregation will reuse it in the later MR.

Next MRs: ReaggregationWorker, ConsistencyWorker, StageAggregationWorker. The flag is enabled only after all four are merged.

Aggregation#estimated_next_run_at still compares against the instance-wide earliest run. It's only a UI estimate, so it's left for a follow-up.

Database

analytics_cycle_analytics_aggregations has about 17,500 enabled rows on GitLab.com, and GitLab.com has about 250 organizations. The table has no organization_id, so load_batch joins namespaces to filter by organization.

Query Plan Result
Aggregation.load_batch(current_time, organization:), default organization (organization_id = 1) https://console.postgres.ai/gitlab/gitlab-production-main/sessions/74891/commands/179077 Walks ca_aggregations_last_incremental_run_at with a namespaces_pkey lookup per row, stops at 100 rows. 1,049 buffers. 498 ms, of which 484 ms is cold-cache I/O.
Same, organization without aggregations (organization_id = 99999999) https://console.postgres.ai/gitlab/gitlab-production-main/sessions/74891/commands/179082 Starts from index_namespaces_on_organization_id and never reads the aggregations. 14 buffers, 3.6 ms.
Aggregation.load_batch(current_time, :last_consistency_check_updated_at, organization:), used by the consistency worker in a later MR https://console.postgres.ai/gitlab/gitlab-production-main/sessions/74891/commands/179078 Same shape as the first row. 606 buffers, 324 ms, of which 308 ms is cold-cache I/O.

The scheduler does not look up which organizations have aggregations. That query reads every enabled aggregation and its namespace: 17,517 namespaces_pkey lookups, about 123,000 buffers and 10.5 seconds, to find 12 organizations (https://console.postgres.ai/gitlab/gitlab-production-main/sessions/74891/commands/179079). Instead, the scheduler queues a job for every organization with Organizations::Organization.each_batch, and the job for an organization without aggregations costs the 14-buffer query above.

SQL

load_batch with an organization:

SELECT "analytics_cycle_analytics_aggregations".* FROM ((SELECT "analytics_cycle_analytics_aggregations".* FROM "analytics_cycle_analytics_aggregations" INNER JOIN "namespaces" ON "namespaces"."id" = "analytics_cycle_analytics_aggregations"."group_id" WHERE (enabled IS TRUE) AND "namespaces"."organization_id" = 1 AND "analytics_cycle_analytics_aggregations"."last_incremental_run_at" IS NULL ORDER BY "analytics_cycle_analytics_aggregations"."last_incremental_run_at" ASC NULLS FIRST LIMIT 100)
UNION ALL
(SELECT "analytics_cycle_analytics_aggregations".* FROM "analytics_cycle_analytics_aggregations" INNER JOIN "namespaces" ON "namespaces"."id" = "analytics_cycle_analytics_aggregations"."group_id" WHERE (enabled IS TRUE) AND "namespaces"."organization_id" = 1 AND "analytics_cycle_analytics_aggregations"."last_incremental_run_at" < '2026-10-29 12:00:00' ORDER BY "analytics_cycle_analytics_aggregations"."last_incremental_run_at" ASC NULLS FIRST LIMIT 100)) analytics_cycle_analytics_aggregations LIMIT 100

References

Screenshots or screen recordings

Not applicable, backend only.

How to set up and validate locally

  1. Make sure VSA aggregations exist for top-level groups in two organizations (Ultimate licence, open Value Stream Analytics for each group once).

  2. In a Rails console, enable the flag and run the cron worker:

    Feature.enable(:vsa_aggregation_per_organization)
    Analytics::CycleAnalytics::IncrementalWorker.new.perform
  3. In log/sidekiq.log, check that there is one Analytics::CycleAnalytics::OrganizationIncrementalWorker job per organization, each with meta.organization_id set.

  4. Disable the flag and run the worker again. No new jobs are scheduled, and last_incremental_run_at is updated for the aggregations of both organizations.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Jatin Parihar

Merge request reports

Loading
Loading