Fix integration propagation timeout with simplified batching

What does this MR do and why?

The integration_propagation_simplified_batching feature flag was rolled out to fix statement timeouts when propagating integration settings across large group hierarchies. After enabling the flag, PG::QueryCanceled (statement timeout) errors continued to occur in PropagateIntegrationWorker.

Root cause

The flag-enabled path batched all descendant projects in a single scope:

Project.in_namespace(namespace_ids) # WHERE namespace_id IN (... hundreds of ids ...)

each_batch orders by primary key, so its boundary query is:

SELECT projects.id FROM projects WHERE namespace_id IN (...) ORDER BY projects.id ASC LIMIT 1

With a large namespace_id IN (...) list and ORDER BY id LIMIT 1, PostgreSQL scans projects_pkey in id order and filters each row against the namespace list, expecting an early match. On large hierarchies the matching rows are sparse/high-id, so it scans a huge portion of the index before the first match — and times out.

Fix

Batch each namespace separately, so every boundary query becomes:

SELECT projects.id FROM projects WHERE namespace_id = ? ORDER BY id ASC LIMIT 1

which uses the index_projects_on_namespace_id_and_id (namespace_id, id) composite index — the fast, bounded plan. PropagateIntegrationProjectWorker re-applies all filters, so the broader batch scope remains safe and idempotent.

Trade-off: at least one worker job is enqueued per descendant namespace with projects (higher Sidekiq job volume).

Database review

Old query

SELECT "projects"."id"
FROM "projects"
WHERE "projects"."namespace_id" IN (/* 9000+ project IDs */)
ORDER BY "projects"."id" ASC
LIMIT 1;

Query plan: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/52066/commands/153387

 Limit  (cost=22.77..44.60 rows=1 width=4) (actual time=97.656..97.658 rows=1 loops=1)
   Buffers: shared hit=989 read=5696
   ->  Index Scan using projects_pkey on public.projects  (cost=22.77..6528953.16 rows=299080 width=4) (actual time=97.654..97.655 rows=1 loops=1)
         Filter: (projects.namespace_id = ANY ('{1, 2, 3, 4, 5, .....}'::integer[]))
         Rows Removed by Filter: 6938
         Buffers: shared hit=989 read=5696
Settings: effective_cache_size = '472585MB', jit = 'off', random_page_cost = '1.5', work_mem = '230MB', seq_page_cost = '4'

Time: 253.657 ms  
  - planning: 155.955 ms  
  - execution: 97.702 ms  
    - I/O read: N/A  
    - I/O write: N/A  
  
Shared buffers:  
  - hits: 989 (~7.70 MiB) from the buffer pool  
  - reads: 5696 (~44.50 MiB) from the OS file cache, including disk I/O  
  - dirtied: 0  
  - writes: 0  

New query

SELECT projects.id
FROM   projects
WHERE  namespace_id = 9970
ORDER  BY projects.id ASC
LIMIT  1; 

Query plan https://console.postgres.ai/gitlab/gitlab-production-main/sessions/52066/commands/153377

 Limit  (cost=0.56..0.67 rows=1 width=4) (actual time=3.543..3.544 rows=1 loops=1)
   Buffers: shared read=6
   ->  Index Only Scan using index_projects_on_namespace_id_and_id on public.projects  (cost=0.56..4.16 rows=34 width=4) (actual time=3.542..3.542 rows=1 loops=1)
         Index Cond: (projects.namespace_id = 9970)
         Heap Fetches: 1
         Buffers: shared read=6
Settings: seq_page_cost = '4', effective_cache_size = '472585MB', jit = 'off', random_page_cost = '1.5', work_mem = '230MB'

Time: 14.101 ms
  - planning: 10.525 ms
  - execution: 3.576 ms
    - I/O read: N/A
    - I/O write: N/A

Shared buffers:
  - hits: 0 from the buffer pool
  - reads: 6 (~48.00 KiB) from the OS file cache, including disk I/O
  - dirtied: 0
  - writes: 0

References

• Rollout issue: #599108 (closed)
• Feature issue: #591182 (closed)