Cap Value Stream Analytics stages per top-level namespace

Brandon Labuschagnerequested to merge
636175-vsa-hierarchy-stage-limit into master

What does this MR do and why?

Rejects a new Value Stream Analytics stage once its top-level namespace hierarchy (the root group or personal namespace with all of its subgroups and project namespaces) already holds 750 stages. The check runs for new stages only, counts persisted stages with the same hierarchy lookup the aggregation crons use plus the unsaved stages submitted in the same value stream save, minus the stages that save removes, and sits behind the vsa_hierarchy_stage_limit feature flag (gitlab_com_derisk, actor: root namespace). Persisted stages are skipped because the value stream services save with an explicit :create validation context, which re-runs create validations on every nested stage; editing or removing stages in a hierarchy at the cap keeps working.

The count-then-insert check is not atomic, so two concurrent saves can land a few stages over the cap. That matches the existing MAX_STAGES_PER_VALUE_STREAM and MAX_VALUE_STREAMS_PER_NAMESPACE checks and is acceptable for a protective limit; the cap is there to stop unbounded growth, not to be exact.

The existing limits (50 value streams per namespace, 15 stages per value stream) reset at every subgroup and project namespace, so one hierarchy can hold an unbounded number of stages. Every new stage schedules a full backfill of the whole top-level group, and every distinct stage adds rows to each batch of the instance-wide aggregation crons. One Reporter can therefore fill the shared crons with one group's work.

750 blocks no current customer: the largest hierarchy on GitLab.com holds 608 stages (second 587, third 505), and 750 equals the existing per-namespace ceiling of 50 × 15. Numbers from Database Lab are in the issue.

Not in this MR: skipping the backfill for a stage whose stage_event_hash_id is already aggregated in the hierarchy (the second MR 2 item on the issue), and the per-run cron budgets (MR 3).

References

Database

One new query, run once per stage on create when the flag is enabled. It reuses the traversal_ids @> '{root_id}' lookup that Stage.distinct_stages_within_hierarchy already runs in the crons; only the outer part changes from DISTINCT ON to COUNT(*).

SELECT COUNT(*)
FROM "analytics_cycle_analytics_group_stages"
WHERE "analytics_cycle_analytics_group_stages"."group_id" IN (
  SELECT namespaces.traversal_ids[array_length(namespaces.traversal_ids, 1)] AS id
  FROM "namespaces"
  WHERE (traversal_ids @> ('{9970}'))
)

Plan for gitlab-org (9970), the hierarchy with the most namespaces on GitLab.com (10,931) and the second-most stages (587). Database Lab gitlab-production-main, Joe command 558993 series, 2026-11-02.

Warm cache: 42 ms 
 Aggregate  (cost=14984.56..14984.57 rows=1 width=8) (actual time=39.109..39.114 rows=1 loops=1)
   Buffers: shared hit=11097
   ->  Hash Join  (cost=13477.17..14933.12 rows=20575 width=0) (actual time=30.740..39.074 rows=587 loops=1)
         Hash Cond: (analytics_cycle_analytics_group_stages.group_id = namespaces.traversal_ids[array_length(namespaces.traversal_ids, 1)])
         Buffers: shared hit=11097
         ->  Index Only Scan using index_group_stages_on_group_id_group_value_stream_id_and_name on public.analytics_cycle_analytics_group_stages  (cost=0.29..1091.04 rows=41150 width=8) (actual time=0.031..5.148 rows=42103 loops=1)
               Heap Fetches: 2927
               Buffers: shared hit=1090
         ->  Hash  (cost=13379.97..13379.97 rows=7753 width=28) (actual time=30.653..30.655 rows=10931 loops=1)
               Buckets: 16384  Batches: 1  Memory Usage: 903kB
               Buffers: shared hit=10007
               ->  HashAggregate  (cost=13302.44..13379.97 rows=7753 width=28) (actual time=25.914..27.177 rows=10931 loops=1)
                     Group Key: namespaces.traversal_ids[array_length(namespaces.traversal_ids, 1)]
                     Batches: 1  Memory Usage: 1425kB
                     Buffers: shared hit=10007
                     ->  Bitmap Heap Scan on public.namespaces  (cost=395.84..13283.05 rows=7753 width=28) (actual time=5.651..22.577 rows=10931 loops=1)
                           Recheck Cond: (namespaces.traversal_ids @> '{9970}'::integer[])
                           Heap Blocks: exact=9803
                           Buffers: shared hit=10007
                           ->  Bitmap Index Scan on index_namespaces_on_traversal_ids  (cost=0.00..393.90 rows=7753 width=0) (actual time=4.278..4.279 rows=10935 loops=1)
                                 Index Cond: (namespaces.traversal_ids @> '{9970}'::integer[])
                                 Buffers: shared hit=204

Time: 42.391 ms (planning 2.789 ms, execution 39.602 ms)
Shared buffers: hits 11097 (~86.70 MiB), reads 0
Cold cache on the clone: 11.5 s, all I/O

Same plan. The first run on the freshly reset clone read 10,358 buffers (~81 MiB) from disk, 11.18 s of the 11.50 s. That is the 9,803 namespaces heap pages of the gitlab-org hierarchy, which on production are hot: the same lookup runs in IncrementalWorker every 10 minutes for every enabled hierarchy.

For a hierarchy of median size (6 stages, a handful of namespaces) the query is a few index lookups.

How to set up and validate locally

  1. Enable the flag and lower the cap in a Rails console (the constant is 750; lower it so the check is reachable with a few stages):

    Feature.enable(:vsa_hierarchy_stage_limit)
Analytics::CycleAnalytics::Stage.send(:remove_const, :MAX_STAGES_PER_HIERARCHY)
Analytics::CycleAnalytics::Stage.const_set(:MAX_STAGES_PER_HIERARCHY, 2)

  2. In a licensed group, open Analyze > Value stream analytics, create a value stream with two custom stages, and save.

  3. In a subgroup or a project of that group, create another value stream with one custom stage. The save fails with Maximum number of stages per top-level namespace exceeded.

  4. Create a value stream with a stage in an unrelated top-level group. It saves.

  5. Feature.disable(:vsa_hierarchy_stage_limit) and repeat step 3. It saves.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code

Edited by Brandon Labuschagne

Merge request reports