Draft: [skip ci] Backfill the tracked context quota of root namespaces

What does this MR do and why?

This MR adds a batched background migration (BBM) that grandfathers the tracked security context quota of existing root namespaces on GitLab.com.

Before we enforce the root namespace quota (!265368), every root namespace that tracks contexts today must get a quota that covers its current usage. Otherwise the GitLab.com default would block it when enforcement starts. The default is 2 today, and #627145 agreed to raise it to 25 in a separate MR.

For each root namespace with tracked contexts, the BBM sets namespace_settings.security_tracked_context_quota to:

quota = tracked contexts + headroom
headroom = min(250,000, max(25, round(6 * monthly growth)))
monthly growth = tracked contexts created in the last 3 months / 3
  • Only contexts in the tracked state count, including the default branch context of each project.
  • The BBM never lowers a quota. If a root namespace already has a higher quota, the BBM keeps it.
  • Root namespaces without tracked contexts get no explicit quota. They use the application default.
  • The BBM runs on GitLab.com only (Gitlab.com_except_jh?). On other instances, no quota applies when the setting is empty, so an explicit quota would start enforcement.

We agreed the formula and the headroom cap by lazy consensus on the confidential issue #627145.

Related to #556012 and &20475. Depends on !265368. The #556012 description still describes an organization backfill. We will update it for root namespaces.

How it works

The BBM iterates all namespaces by primary key, like BackfillMcpServerEnabled. It does not use scope_to. It uses the primary key batching strategy. namespaces has a single integer primary key, so cursor batching would give the same batches.

For each sub-batch of 2,500 namespaces:

  1. Select the ids of the root namespaces (parent_id IS NULL) in the sub-batch (main database). If the sub-batch has no root namespaces, go to the next sub-batch.
  2. Count tracked contexts per root namespace on the sec database. traversal_ids[1] is the root namespace. A traversal_ids[1] predicate cannot use index_security_project_tracked_contexts_on_traversal_ids_id. The query uses a range on the whole array instead (traversal_ids >= '{min}' AND traversal_ids < '{max + 1}').
  3. Ignore rows whose first traversal id is not a root namespace in the sub-batch.
  4. Write all quotas of the sub-batch with one INSERT ... ON CONFLICT (namespace_id) DO UPDATE on namespace_settings. The update applies only when the existing quota is empty or lower.

About 2 in 5 namespaces are root namespaces, so a sub-batch of 2,500 namespaces holds about 1,000 root namespaces. Old namespace ids are denser. The S2 sub-batch is all root namespaces.

There is no cross-database query and no cross-database transaction. The BBM depends on the traversal_ids backfill (queued in 20260904100831, finalized in 20261015234740), so traversal_ids is complete.

tables_to_check_for_vacuum :namespace_settings makes the BBM pause for autovacuum on the table it writes, not on namespaces.

Batching design

We considered three ways to select the root namespaces:

Option Batching query Decision
scope_to with parent_id IS NULL Walks namespaces_pkey and filters on parent_id. In an earlier plan of the next batch query (Joe command id 539824), one batch had Rows Removed by Filter: 36882. Each sub-batch boundary also sorts up to one batch of index entries. The planner does not use the id order of index_namespaces_on_parent_id_and_id for parent_id IS NULL. Rejected. The BBM guide allows scope_to only when the batching query filters out no rows and uses an index-only scan. The cost of a batch also depends on how many root namespaces its id range holds, and some ranges hold very few.
scope_to with a new partial index on namespaces (id) WHERE parent_id IS NULL Index-only scan. Rejected. The index is permanent and adds write cost on a large, busy table, for a migration that runs once.
Primary key batching, with the filter in each sub-batch (this MR) Primary key ranges only (M3 and M1). The batching queries have the same cost in each batch. Chosen. The guide shows this pattern in Apply selection without scope_to.

The chosen option iterates over all namespaces, not only root namespaces. Because about 2 in 5 namespaces are root namespaces, the BBM runs about 2.5 times as many batches as a BBM over root namespaces only. Sub-batches without root namespaces skip the gitlab_sec query. The batch optimizer can increase the batch size.

Rollout

Enforcement stays off until this BBM has finished on GitLab.com. We enable the security_tracked_context_quota_enforcement flag (!265368) only after that.

The BBM takes one snapshot. Root namespaces that start to track contexts after their batch ran get the default quota. Growth after the run uses up the 6 month headroom. If enforcement starts much later, we can queue the same job again. It never lowers a quota.

A finalize migration follows in a later milestone, after a required stop.

Before database review

  • Add the output of the migration up (with Gitlab.com_except_jh? stubbed to true) and down.

Production estimates

Ratios from the sec and main databases (postgres.ai). Row 1 divides the 2026-10-27 count of root namespaces with tracked contexts by the 2026-10-20 root namespace count. Rows 2 and 3 are from 2026-10-27. Row 4 is from 2026-11-06.

Item Value
Root namespaces with tracked contexts (rows written), as a share of all root namespaces about 1 in 7
Root namespaces with growth headroom above 25, as a share of rows written about 1 in 300
Root namespaces at the 250,000 headroom cap none
Root namespaces as a share of all namespaces about 2 in 5

The expected run time and the absolute counts are in an internal note on this MR.

Queries

We ran the plans through the postgres.ai Joe CLI on 2026-11-06 at the BBM batch size (50,000) and sub-batch size (2,500). The whole-table counts are in the internal note, with their Joe command ids.

<id_a> to <id_f> stand for real namespace ids. <id_a> is the first id of the batch and of its first sub-batch. <id_c> is the first id of the second sub-batch. <id_d> is the last id of the batch. <id_e> is the first id of the S2 sub-batch and <id_f> is its last id plus 1. The real next-batch query repeats the id >= <id_a> condition. M3 leaves out the repeat. M1 shows that the repeat only adds the same condition to the index condition.

M4 ran on a clone where Joe first added the security_tracked_context_quota column, because the column does not exist in production yet. The BBM sends a VALUES list. M4 uses INSERT ... SELECT over the first 1,560 root namespaces of the S2 sub-batch, the same number of rows that S2 returns, to get the same conflict path without real ids. Joe runs it in a rolled-back transaction.

Joe returned the plans inline and produced no console.postgres.ai links.

Query Project Joe command id Time Buffer hits Reads (OS cache or disk) Dirtied Written
M3 Next batch boundary (batching strategy, batch size 50,000) gitlab-production-main 572946 50.134 ms 10176 (79.50 MiB) 63 (504.00 KiB) 0 0
M1 Sub-batch boundary (each_batch OFFSET query) gitlab-production-main 572949 2.907 ms 1106 (8.60 MiB) 0 0 0
M2 Root namespace ids of the sub-batch (pluck) gitlab-production-main 572952 19.006 ms 425 (3.30 MiB) 16 (128.00 KiB) 0 0
S1 Usage per root namespace, slowest sub-batch found gitlab-production-sec 572945 5.939 s 157050 (1.20 GiB) 40596 (317.20 MiB) 951 (7.40 MiB) 0
S2 Usage per root namespace, dense sub-batch (old namespace ids) gitlab-production-sec 572954 5.184 s 4532 (35.40 MiB) 12568 (98.20 MiB) 242 (1.90 MiB) 0
M4 Upsert of namespace_settings (1,560 rows, S2 sub-batch) gitlab-production-main 572997 2.701 s 28752 (224.60 MiB) 3686 (28.80 MiB) 2215 (17.30 MiB) 2 (16.00 KiB)

M3 ran first in the clone session and found most of its pages in the buffer pool. M1 ran after it on the same namespaces_pkey pages. M2 reads index_namespaces_on_parent_id_and_id. The S1 plan is in the internal note.

On the cold clones, S1, S2 and M4 take longer than the 1 s guideline for queries in background migrations. Most of that time is I/O read time. Without the I/O read time, the execution of S1 takes about 0.4 s, S2 about 0.1 s and M4 about 0.1 s.

M3. Next batch boundary (batching strategy, batch size 50,000)

Project: gitlab-production-main. Joe command id: 572946.

SELECT namespaces.id FROM namespaces WHERE namespaces.id >= <id_a> ORDER BY namespaces.id ASC LIMIT 1 OFFSET 50000
 Limit  (cost=1283.53..1283.56 rows=1 width=4) (actual time=48.308..48.309 rows=1 loops=1)
   Buffers: shared hit=10176 read=63
   I/O Timings: read=36.490 write=0.000
   ->  Index Only Scan using namespaces_pkey on public.namespaces  (cost=0.57..329198.51 rows=12829605 width=4) (actual time=0.659..46.492 rows=50001 loops=1)
         Index Cond: (namespaces.id >= <id_a>)
         Heap Fetches: 1260
         Buffers: shared hit=10176 read=63
         I/O Timings: read=36.490 write=0.000
Settings: jit = 'off', random_page_cost = '1.5', seq_page_cost = '4', effective_cache_size = '472585MB', work_mem = '230MB'
Query ID: 8589541813036077055

Joe stats:

Time: 50.134 ms
  - planning: 1.794 ms
  - execution: 48.340 ms
    - I/O read: 36.490 ms
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 10176 (~79.50 MiB) from the buffer pool
  - reads: 63 (~504.00 KiB) from the OS file cache, including disk I/O
  - dirtied: 0
  - writes: 0

M1. Sub-batch boundary (each_batch OFFSET query)

Project: gitlab-production-main. Joe command id: 572949.

SELECT namespaces.id FROM namespaces WHERE namespaces.id BETWEEN <id_a> AND <id_d> AND namespaces.id >= <id_a> ORDER BY namespaces.id ASC LIMIT 1 OFFSET 2500
 Limit  (cost=99.08..99.12 rows=1 width=4) (actual time=1.009..1.010 rows=1 loops=1)
   Buffers: shared hit=1106
   I/O Timings: read=0.000 write=0.000
   ->  Index Only Scan using namespaces_pkey on public.namespaces  (cost=0.57..2113.43 rows=53619 width=4) (actual time=0.036..0.914 rows=2501 loops=1)
         Index Cond: ((namespaces.id >= <id_a>) AND (namespaces.id <= <id_d>) AND (namespaces.id >= <id_a>))
         Heap Fetches: 49
         Buffers: shared hit=1106
         I/O Timings: read=0.000 write=0.000
Settings: random_page_cost = '1.5', seq_page_cost = '4', effective_cache_size = '472585MB', work_mem = '230MB', jit = 'off'
Query ID: -1352958676737478102

Joe stats:

Time: 2.907 ms
  - planning: 1.871 ms
  - execution: 1.036 ms
    - I/O read: 0.000 ms
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 1106 (~8.60 MiB) from the buffer pool
  - reads: 0 from the OS file cache, including disk I/O
  - dirtied: 0
  - writes: 0

M2. Root namespace ids of the sub-batch (pluck)

Project: gitlab-production-main. Joe command id: 572952. This sub-batch has 1,039 root namespaces.

SELECT namespaces.id FROM namespaces WHERE namespaces.id BETWEEN <id_a> AND <id_d> AND namespaces.id >= <id_a> AND namespaces.id < <id_c> AND namespaces.parent_id IS NULL
 Index Only Scan using index_namespaces_on_parent_id_and_id on public.namespaces  (cost=0.57..79.44 rows=1456 width=4) (actual time=7.866..17.090 rows=1039 loops=1)
   Index Cond: ((namespaces.parent_id IS NULL) AND (namespaces.id >= <id_a>) AND (namespaces.id <= <id_d>) AND (namespaces.id >= <id_a>) AND (namespaces.id < <id_c>))
   Heap Fetches: 21
   Buffers: shared hit=425 read=16
   I/O Timings: read=16.590 write=0.000
Settings: seq_page_cost = '4', effective_cache_size = '472585MB', work_mem = '230MB', jit = 'off', random_page_cost = '1.5'
Query ID: -1156189017433651835

Joe stats:

Time: 19.006 ms
  - planning: 1.844 ms
  - execution: 17.162 ms
    - I/O read: 16.590 ms
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 425 (~3.30 MiB) from the buffer pool
  - reads: 16 (~128.00 KiB) from the OS file cache, including disk I/O
  - dirtied: 0
  - writes: 0

S2. Usage per root namespace, dense sub-batch (old namespace ids)

Project: gitlab-production-sec. Joe command id: 572954. Every namespace in this sub-batch is a root namespace.

SELECT traversal_ids[1] AS root_namespace_id, COUNT(*) AS tracked_count, COUNT(*) FILTER (WHERE created_at >= '2026-08-06 13:00:00.000000') AS recent_count FROM security_project_tracked_contexts WHERE traversal_ids >= '{<id_e>}' AND traversal_ids < '{<id_f>}' AND state = 2 GROUP BY traversal_ids[1]
 HashAggregate  (cost=30703.30..30915.51 rows=21221 width=24) (actual time=5181.863..5182.216 rows=1560 loops=1)
   Group Key: security_project_tracked_contexts.traversal_ids[1]
   Batches: 1  Memory Usage: 665kB
   Buffers: shared hit=4532 read=12568 dirtied=242
   WAL: records=346 fpi=242 bytes=1969458
   I/O Timings: read=5081.968 write=0.000
   ->  Index Scan using index_security_project_tracked_contexts_on_traversal_ids_id on public.security_project_tracked_contexts  (cost=0.56..30489.10 rows=21420 width=16) (actual time=2.717..5167.226 rows=19319 loops=1)
         Index Cond: ((security_project_tracked_contexts.traversal_ids >= '{<id_e>}'::bigint[]) AND (security_project_tracked_contexts.traversal_ids < '{<id_f>}'::bigint[]))
         Index Searches: 1
         Filter: (security_project_tracked_contexts.state = 2)
         Rows Removed by Filter: 7
         Buffers: shared hit=4532 read=12568 dirtied=242
         WAL: records=346 fpi=242 bytes=1969458
         I/O Timings: read=5081.968 write=0.000
Settings: work_mem = '100MB', random_page_cost = '1.5', seq_page_cost = '4', effective_cache_size = '338688MB', jit = 'off'
Query ID: 6740866070236560504

Joe stats:

Time: 5.184 s
  - planning: 1.115 ms
  - execution: 5.183 s
    - I/O read: 5.082 s
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 4532 (~35.40 MiB) from the buffer pool
  - reads: 12568 (~98.20 MiB) from the OS file cache, including disk I/O
  - dirtied: 242 (~1.90 MiB)
  - writes: 0

M4. Upsert of namespace_settings (1,560 rows, S2 sub-batch)

Project: gitlab-production-main. Joe command id: 572997.

INSERT INTO namespace_settings (namespace_id, security_tracked_context_quota, created_at, updated_at) SELECT id, 26, '2026-11-06 13:30:00.000000', '2026-11-06 13:30:00.000000' FROM namespaces WHERE parent_id IS NULL AND id >= <id_e> ORDER BY id LIMIT 1560 ON CONFLICT (namespace_id) DO UPDATE SET security_tracked_context_quota = EXCLUDED.security_tracked_context_quota, updated_at = EXCLUDED.updated_at WHERE namespace_settings.security_tracked_context_quota IS NULL OR namespace_settings.security_tracked_context_quota < EXCLUDED.security_tracked_context_quota
 Insert on public.namespace_settings  (cost=0.57..379.82 rows=0 width=0) (actual time=2615.174..2615.176 rows=0 loops=1)
   Buffers: shared hit=28752 read=3686 dirtied=2215 written=2
   WAL: records=7399 fpi=2205 bytes=18131771
   I/O Timings: read=2511.428 write=0.067
   ->  Subquery Scan on "*SELECT*"  (cost=0.57..379.82 rows=1560 width=567) (actual time=1.286..1155.758 rows=1560 loops=1)
         Buffers: shared hit=170 read=1398
         I/O Timings: read=1145.202 write=0.000
         ->  Limit  (cost=0.57..364.22 rows=1560 width=72) (actual time=1.279..1153.861 rows=1560 loops=1)
               Buffers: shared hit=170 read=1398
               I/O Timings: read=1145.202 write=0.000
               ->  Index Scan using namespaces_pkey on public.namespaces  (cost=0.57..8310862.80 rows=35651540 width=72) (actual time=1.277..1153.341 rows=1560 loops=1)
                     Index Cond: (namespaces.id >= <id_e>)
                     Filter: (namespaces.parent_id IS NULL)
                     Buffers: shared hit=170 read=1398
                     I/O Timings: read=1145.202 write=0.000
Settings: jit = 'off', random_page_cost = '1.5', seq_page_cost = '4', effective_cache_size = '472585MB', work_mem = '230MB'
Query ID: -1343575597278063016

Joe stats:

Time: 2.701 s
  - planning: 85.256 ms
  - execution: 2.615 s
    - I/O read: 2.511 s
    - I/O write: 0.067 ms

Shared buffers:
  - hits: 28752 (~224.60 MiB) from the buffer pool
  - reads: 3686 (~28.80 MiB) from the OS file cache, including disk I/O
  - dirtied: 2215 (~17.30 MiB)
  - writes: 2 (~16.00 KiB)

From the JSON plan that Joe returned for M4:

Conflict Resolution: UPDATE
Conflict Arbiter Indexes: namespace_settings_pkey
Tuples Inserted: 0
Conflicting Tuples: 1560
Rows Removed by Conflict Filter: 0

All 1,560 rows already had a namespace_settings row, so M4 measured the update path. M4 also reads the 1,560 root namespaces from namespaces (about 1.2 s cold). The BBM sends the ids in a VALUES list and does not run that part. The insert path is covered by the spec for a user namespace without a settings row.

How to set up and validate locally

The migration does not queue the BBM outside GitLab.com, so step 3 runs the job directly.

  1. Check out the branch and run bundle exec rails db:migrate.

  2. Create a root group with a project, and create 12 or fewer tracked contexts for it on the sec database. With 12 or fewer, the minimum headroom of 25 applies.

  3. In the Rails console, run the migration job for the namespace id range:

    Gitlab::BackgroundMigration::BackfillRootNamespaceSecurityTrackedContextQuota.new(
      start_id: Namespace.minimum(:id), end_id: Namespace.maximum(:id),
      batch_table: :namespaces, batch_column: :id, sub_batch_size: 2_500, pause_ms: 0,
      connection: ApplicationRecord.connection
    ).perform
  4. Check that NamespaceSetting.find_by(namespace_id: <root id>).security_tracked_context_quota is the tracked count plus 25.

Rollback

The down method deletes the queued BBM. The BBM does not delete data. To remove the quotas it wrote, set security_tracked_context_quota back to NULL for the affected root namespaces. Do this only before enforcement is enabled. After Support starts to raise quotas (#585974), we cannot tell those quotas apart from the quotas that this BBM wrote.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Gregory Havenga

Merge request reports

Loading
Loading