Run VSA incremental aggregation one organization per job
What does this MR do and why?
On Cells, every Value Stream Analytics aggregation job must work for one organization only. Today Analytics::CycleAnalytics::IncrementalWorker processes the aggregations of every organization in one cron job, with no organization context. A large organization can use up the 200-second runtime budget of the others.
This is the first of four MRs, one per VSA aggregation cron worker. It covers the incremental worker.
- Behind the
vsa_aggregation_per_organizationflag (gitlab_com_derisk, default off),IncrementalWorkerschedules oneOrganizationIncrementalWorkerjob per organization, withorganization:in the job context. Organizations without aggregations finish after one cheap query (see Database). With the flag off,IncrementalWorkerruns the same loop inline for the whole instance, as it does today. OrganizationIncrementalWorkerloads only the aggregations of its organization (Aggregation.load_batch(..., organization:)), with its own runtime limit.OrganizationIncrementalWorkeris a separate class becauseCronjobQueue.context_for_argumentsreplaces the batch context of a cron worker's own jobs withcaller_id: "Cronjob", which would drop the organization.- Removes the
Scalability/CronWorkerContextdisable fromIncrementalWorker. - The new worker uses
data_consistency :sticky, likeStageAggregationWorker, andretry: false, like the cron worker it replaces, because the next cron run picks up any leftover work. Parentablegets afor_organizationscope.StageAggregationwill reuse it in the later MR.
Next MRs: ReaggregationWorker, ConsistencyWorker, StageAggregationWorker. The flag is enabled only after all four are merged.
Aggregation#estimated_next_run_at still compares against the instance-wide earliest run. It's only a UI estimate, so it's left for a follow-up.
Database
analytics_cycle_analytics_aggregations has about 17,500 enabled rows on GitLab.com, and GitLab.com has about 250 organizations. The table has no organization_id, so load_batch joins namespaces to filter by organization.
| Query | Plan | Result |
|---|---|---|
Aggregation.load_batch(current_time, organization:), default organization (organization_id = 1) |
https://console.postgres.ai/gitlab/gitlab-production-main/sessions/74891/commands/179077 | Walks ca_aggregations_last_incremental_run_at with a namespaces_pkey lookup per row, stops at 100 rows. 1,049 buffers. 498 ms, of which 484 ms is cold-cache I/O. |
Same, organization without aggregations (organization_id = 99999999) |
https://console.postgres.ai/gitlab/gitlab-production-main/sessions/74891/commands/179082 | Starts from index_namespaces_on_organization_id and never reads the aggregations. 14 buffers, 3.6 ms. |
Aggregation.load_batch(current_time, :last_consistency_check_updated_at, organization:), used by the consistency worker in a later MR |
https://console.postgres.ai/gitlab/gitlab-production-main/sessions/74891/commands/179078 | Same shape as the first row. 606 buffers, 324 ms, of which 308 ms is cold-cache I/O. |
The scheduler does not look up which organizations have aggregations. That query reads every enabled aggregation and its namespace: 17,517 namespaces_pkey lookups, about 123,000 buffers and 10.5 seconds, to find 12 organizations (https://console.postgres.ai/gitlab/gitlab-production-main/sessions/74891/commands/179079). Instead, the scheduler queues a job for every organization with Organizations::Organization.each_batch, and the job for an organization without aggregations costs the 14-buffer query above.
SQL
load_batch with an organization:
SELECT "analytics_cycle_analytics_aggregations".* FROM ((SELECT "analytics_cycle_analytics_aggregations".* FROM "analytics_cycle_analytics_aggregations" INNER JOIN "namespaces" ON "namespaces"."id" = "analytics_cycle_analytics_aggregations"."group_id" WHERE (enabled IS TRUE) AND "namespaces"."organization_id" = 1 AND "analytics_cycle_analytics_aggregations"."last_incremental_run_at" IS NULL ORDER BY "analytics_cycle_analytics_aggregations"."last_incremental_run_at" ASC NULLS FIRST LIMIT 100)
UNION ALL
(SELECT "analytics_cycle_analytics_aggregations".* FROM "analytics_cycle_analytics_aggregations" INNER JOIN "namespaces" ON "namespaces"."id" = "analytics_cycle_analytics_aggregations"."group_id" WHERE (enabled IS TRUE) AND "namespaces"."organization_id" = 1 AND "analytics_cycle_analytics_aggregations"."last_incremental_run_at" < '2026-10-29 12:00:00' ORDER BY "analytics_cycle_analytics_aggregations"."last_incremental_run_at" ASC NULLS FIRST LIMIT 100)) analytics_cycle_analytics_aggregations LIMIT 100References
- #636944
- Instance-wide cron approach: #599101
- Similar organization-context fan-outs: #607102 (closed)
Screenshots or screen recordings
Not applicable, backend only.
How to set up and validate locally
-
Make sure VSA aggregations exist for top-level groups in two organizations (Ultimate licence, open Value Stream Analytics for each group once).
-
In a Rails console, enable the flag and run the cron worker:
Feature.enable(:vsa_aggregation_per_organization) Analytics::CycleAnalytics::IncrementalWorker.new.perform -
In
log/sidekiq.log, check that there is oneAnalytics::CycleAnalytics::OrganizationIncrementalWorkerjob per organization, each withmeta.organization_idset. -
Disable the flag and run the worker again. No new jobs are scheduled, and
last_incremental_run_atis updated for the aggregations of both organizations.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.