Skip saving unchanged code owner approval rules

The code owner rule sync saved every matching rule on every run, even when the rule already matched its CODEOWNERS entry. Each of those saves opened a transaction for two validation reads and wrote nothing, and each rule's users and groups were loaded one rule at a time. This saves only the rules that changed and preloads their users and groups, so a sync that changes nothing runs 6 statements and no transaction for any number of entries. It's behind code_owner_rules_sync_skip_unchanged_saves.

For #639974. Part of &24375.

Detailed context for AI agents

Why

  • The main non-urgent Sidekiq PgBouncer pool (gitlabhq_production_sidekiq) saturates on weekday afternoons. MergeRequestMergeabilityCheckWorker is its largest holder, and most of its held time is idle in transaction: the thread waits about 5 ms for the GVL after each Postgres reply.
  • MergeRequests::SyncCodeOwnerApprovalRules runs after every merge-head diff rebuild (EE::MergeRequests::ReloadMergeHeadDiffService), so on every mergeability check that ends mergeable. The epic puts it at about 10% of that worker's idle-in-transaction time.
  • While the pool is saturated, each primary query in that worker waits about 290 ms for a connection (peak hour 2026-11-12 14:00-15:00 UTC: full checks averaged 73 queries and 22.8 s, 92% of it database time). So every query removed also shortens the job.
  • The same sync runs on every source push (MergeRequests::Refresh::ApprovalService), from SyncCodeOwnerApprovalRulesWorker and from EE::ProtectedBranches::CreateService.

What happened per matching entry

For a rule that already exists and matches its entry, execute did:

  1. rule.users = entry.users and rule.groups = entry.groups: each loads the rule's current members (one query each, per rule) and writes nothing when they match. These associations write on assignment, not on save.
  2. rule.role_approvers = and rule.approvals_required =: in-memory assignments.
  3. rule.save: opens a transaction, runs the name uniqueness check and validate_approval_project_rule (a lookup of approval_project_rules), then commits with no write.

So each matching entry cost 4 reads and one transaction holding 2 of them.

Change

  • rule.save runs only when rule.changed? (dirty tracking on approvals_required and role_approvers). A new rule is already inserted by ApprovalMergeRequestRule.find_or_create_code_owner_rule before this point.
  • The existing rules are loaded with preload_users_and_groups (new scope on ApprovalMergeRequestRule), so the two per-rule member loads become three queries in total.
  • Both are gated by code_owner_rules_sync_skip_unchanged_saves (gitlab_com_derisk, actor: the target project). With the flag off the code path is the same as before.

Why skipping the save is safe:

  • Users and groups are has_and_belongs_to_many, so assignment inserts and deletes join rows immediately. The save never wrote them.
  • The only update callback is before_update :compare_with_project_rule, which sets modified_from_project_rule from overridden?. Code owner rules have no source rule, so overridden? is always false for them.
  • The validations a skipped save would have run are on a rule whose attributes didn't change.
  • role_approvers is an array, so the same roles in a different order count as changed. That only triggers a save that happened before too.

Measurements

Test environment, a sync where nothing changed (entries stubbed, so loading the entries from CODEOWNERS is not included and is unchanged):

Statements Transactions Statements inside transactions
2 entries, before 11 2 4
2 entries, after 6 0 0
20 entries, before 83 20 40
20 entries, after 6 0 0

A local trace of a full mergeability check on a merge request with 58 matching entries showed the same shape: 244 of its 290 queries were in this sync, 4 per entry.

A probe also drifted approvals_required, the role approvers, a removed user and group, an entry with changed owners, and a new plus a removed entry. The sync restored the expected rules in every case.

Specs

ee/spec/services/merge_requests/sync_code_owner_approval_rules_spec.rb:

  • updates a rule whose required approvals or roles drifted,
  • runs no uniqueness check when the rules already match,
  • runs no more queries for three entries than for two,
  • with the flag disabled, saves every rule as before.

That spec, ee/spec/services/ee/merge_requests/reload_merge_head_diff_service_spec.rb and ee/spec/workers/merge_requests/sync_code_owner_approval_rules_worker_spec.rb pass locally (29 examples).

Flag and rollout

  • code_owner_rules_sync_skip_unchanged_saves, gitlab_com_derisk, default off, no changelog. Rollout issue: #639975.
  • With the flag off, the sync runs the same SQL as the service on master, statement for statement (compared for a sync that changes nothing and for one that creates, changes and removes rules), and reads the flag once per sync. The actor, the target project, is already loaded by then, because loading the entries reads it first.
  • Ungated but inert: the new preload_users_and_groups scope, which only the flag-on path calls.

Not in scope

  • Loading the entries themselves (Gitlab::CodeOwners.entries_for_merge_request), which reads the CODEOWNERS blob and looks up owners.

Database queries

Added during database review.

With the flag on, the sync loads the users and groups of the existing code owner rules in three queries, instead of two per rule. The query that loads the rules is unchanged.

The plans are from a postgres.ai clone of production with a cold cache, so almost all of the time is I/O read. They use the code owner rules of two merged merge requests: !265516 (merged) (62 rules) and !267763 (merged) (354 rules).

Query 62 rules 354 rules
1. User join rows 3 rows, 35 buffers, 21.9 ms (plan) 350 rows, 339 buffers, 36.7 ms (plan)
2. Users 2 rows, 8 buffers, 4.6 ms (plan) 17 rows, 76 buffers, 74.4 ms (plan)
3. Group join rows with groups 78 rows, 454 buffers, 103.0 ms (plan) not run

The planner overestimates the rows of query 1 (294,820 estimated for 3, and 1,683,329 for 350). It still uses the unique index on (approval_merge_request_rule_id, user_id).

1. User join rows
SELECT "approval_merge_request_rules_users".*
FROM "approval_merge_request_rules_users"
WHERE "approval_merge_request_rules_users"."approval_merge_request_rule_id" IN (398529857, 398529864, ...)

62 rules:

 Index Scan using index_approval_merge_request_rules_users_1 on public.approval_merge_request_rules_users  (cost=0.58..218788.62 rows=294820 width=28) (actual time=12.134..18.798 rows=3 loops=1)
   Index Cond: (approval_merge_request_rules_users.approval_merge_request_rule_id = ANY ('{398529857,398529864,...}'::bigint[]))
   Buffers: shared hit=22 read=13
   I/O Timings: read=18.627 write=0.000

Time: 21.902 ms
  - planning: 3.060 ms
  - execution: 18.842 ms
    - I/O read: 18.627 ms

354 rules:

 Index Scan using index_approval_merge_request_rules_users_1 on public.approval_merge_request_rules_users  (cost=0.58..1210472.79 rows=1683329 width=28) (actual time=12.425..31.528 rows=350 loops=1)
   Index Cond: (approval_merge_request_rules_users.approval_merge_request_rule_id = ANY ('{402106728,402106729,...}'::bigint[]))
   Buffers: shared hit=306 read=33 dirtied=4
   I/O Timings: read=31.260 write=0.000

Time: 36.728 ms
  - planning: 5.153 ms
  - execution: 31.575 ms
    - I/O read: 31.260 ms
2. Users
SELECT "users".* FROM "users" WHERE "users"."id" IN (12190616, 23402697)

62 rules, 2 distinct users:

 Index Scan using users_pkey on public.users  (cost=0.44..6.56 rows=2 width=1663) (actual time=0.025..0.038 rows=2 loops=1)
   Index Cond: (users.id = ANY ('{12190616,23402697}'::integer[]))
   Buffers: shared hit=8

Time: 4.639 ms
  - planning: 4.530 ms
  - execution: 0.109 ms

354 rules, 17 distinct users:

 Index Scan using users_pkey on public.users  (cost=0.44..54.08 rows=17 width=1663) (actual time=10.064..69.650 rows=17 loops=1)
   Index Cond: (users.id = ANY ('{164827,480804,...}'::integer[]))
   Buffers: shared hit=22 read=54 dirtied=4
   I/O Timings: read=69.287 write=0.000

Time: 74.441 ms
  - planning: 4.712 ms
  - execution: 69.729 ms
    - I/O read: 69.287 ms
3. Group join rows with groups

Rails lists every column of both tables with an alias. They are written as .* here.

SELECT "approval_merge_request_rules_groups".*, "namespaces".*
FROM "approval_merge_request_rules_groups"
LEFT OUTER JOIN "namespaces"
  ON "namespaces"."id" = "approval_merge_request_rules_groups"."group_id"
  AND "namespaces"."type" = 'Group'
WHERE "namespaces"."type" = 'Group'
  AND "approval_merge_request_rules_groups"."approval_merge_request_rule_id" IN (398529857, 398529864, ...)

62 rules:

 Nested Loop  (cost=1.13..748.27 rows=18 width=380) (actual time=14.708..99.881 rows=78 loops=1)
   Buffers: shared hit=376 read=78 dirtied=1
   I/O Timings: read=98.582 write=0.000
   ->  Index Scan using index_approval_merge_request_rules_groups_1 on public.approval_merge_request_rules_groups  (cost=0.56..235.26 rows=143 width=20) (actual time=8.956..25.326 rows=78 loops=1)
         Index Cond: (approval_merge_request_rules_groups.approval_merge_request_rule_id = ANY ('{398529857,398529864,...}'::bigint[]))
         Buffers: shared hit=38 read=26 dirtied=1
         I/O Timings: read=24.445 write=0.000
   ->  Index Scan using namespaces_pkey on public.namespaces  (cost=0.57..3.59 rows=1 width=360) (actual time=0.955..0.955 rows=1 loops=78)
         Index Cond: (namespaces.id = approval_merge_request_rules_groups.group_id)
         Filter: ((namespaces.type)::text = 'Group'::text)
         Buffers: shared hit=338 read=52
         I/O Timings: read=74.138 write=0.000

Time: 102.964 ms
  - planning: 3.009 ms
  - execution: 99.955 ms
    - I/O read: 98.582 ms
Edited by Igor Drozdov

Merge request reports

Loading
Loading