Fix fork merge request pipeline count timeout (!260777) · Merge requests · GitLab.org / GitLab

What does this MR do and why?

Fork merge request pages intermittently return HTTP 500 for a viewer who can read pipelines on the target project but not on the source (fork) project. The page fails with PG::QueryCanceled: canceling statement due to statement timeout on one COUNT(*) issued by Ci::PipelinesForMergeRequestFinder. That count backs the pipeline-count badge on the MR page (@number_of_pipelines) and the count in the pipelines JSON action.

This reproduces on gitlab.com today, including on MRs with zero matching pipelines (for example omnibus-gitlab!9563 (closed) and omnibus-gitlab!9711 (closed)). Because it happens with no matching rows, this is a query-plan problem, not a data-volume problem.

This MR makes the target-only query a single indexed lookup, and it ships behind the ci_pipelines_for_mr_single_project_scope feature flag, disabled by default. The returned pipelines are identical in every permission case whether the flag is on or off.

Root cause

For a target-only viewer, the finder generated the merge-request-event predicate as the redundant pair project_id IN (source, target) AND project_id = target. That pair makes the PostgreSQL planner mis-estimate selectivity, so it picks the broad index index_ci_pipelines_on_project_id_and_status_and_config_source instead of the selective partial index p_ci_pipelines_merge_request_id_idx. It then heap-visits every pipeline of the (large) target project, across every partition of p_ci_pipelines (the query has no partition_id, so there is no partition pruning). In the reproduction that is 43,674 rows removed by filter in the default partition alone, about 426 MiB read. Cold, that takes about 56 seconds, well past the 15 second statement timeout.

The redundant pair came from the shared scope Ci::Pipeline.triggered_by_merge_request, which hardcoded project: [source_project, target_project] (an IN list). The finder then re-scoped it with .for_project(target) (an equality). The original production code wrapped .for_project(target) around a from_union(...), which produces the same redundant pair on the surviving arm.

Either predicate alone is fine: project_id = target alone, or project_id IN (source, target) alone (the both-projects-readable path, which is unchanged and already fast). Only the redundant combination defeats the planner.

Note for reviewers reading the plan: the UNION branch arm was not the cost. PostgreSQL already prunes it to a One-Time Filter: false (it is provably empty under project_id = target), and the de-duplication Sort processed 0 rows. The entire cost is the index misselection.

The fix

Scope the merge-request-event pipelines to the single authorized project with an equality predicate (project_id = <project>) instead of the redundant IN-plus-equality pair. To do this, the shared scope Ci::Pipeline.triggered_by_merge_request now accepts the project(s) to scope to. It defaults to both projects, so callers that do not pass a single project generate identical SQL to before: MergeRequest#all_pipelines (which passes no user and keeps the IN list) and the both-readable path. Only the two partial-permission paths change. The finder passes the single authorized project for those two paths, which yields project_id = <project> and lets the planner use the selective merge_request_id index via a BitmapAnd.

Scope of change:

Branch pipelines only ever belong to the source project. For a target-only viewer the branch arm and the UNION are dropped; the query is a single equality-scoped lookup.
For a source-only viewer the UNION is kept (both arms are relevant), and each arm is now scoped by equality to the source project.
The both-readable path (all) and the no-permission path (none) are unchanged.
A non-persisted (unsaved) fork MR with partial permission is also tidied. It previously stacked a redundant project_id = source AND project_id = <project>. It now returns the source branch pipelines for a source-readable viewer and nothing for a target-only viewer (branch pipelines are source-only). Same results.
authorizing_project is intentionally not reused inside execute. It returns one source-preferring project for the GraphQL resolver's single-project finder call, whereas execute must distinguish zero, one, or both readable projects. The resolver (app/graphql/resolvers/merge_request_pipelines_resolver.rb) benefits from this change automatically and its fork-MR specs pass unchanged.

Behaviour is unchanged: the returned pipelines are identical in every permission case (both, source-only, target-only, neither, nil actor, non-persisted). This is a pure query-shape change.

Files changed:

app/finders/ci/pipelines_for_merge_request_finder.rb
app/models/ci/pipeline.rb (the scope now takes an optional projects argument, defaulting to both projects)
spec/finders/ci/pipelines_for_merge_request_finder_spec.rb
config/feature_flags/gitlab_com_derisk/ci_pipelines_for_mr_single_project_scope.yml

Feature flag

This change is behind the ci_pipelines_for_mr_single_project_scope feature flag, a gitlab_com_derisk flag that ships disabled (default_enabled: false). The actor is the merge request's target project.

With the flag disabled, the finder runs the exact previous production query, which scopes the full union by project after building it. This is the slow query that times out.
With the flag enabled, the two partial-permission paths scope the merge-request-event pipelines to the single authorized project by equality (project_id = <project>). This is the fast query.

The returned pipelines are identical either way, and the finder and resolver specs assert both flag states. Rollout is tracked in #634103: enable on GitLab.com, confirm the 500s stop with no new statement timeouts, then remove the flag and the old code path.

SQL before and after

Target-only viewer; source value 10 is merge_request_event; ids are shown as placeholders.

Before (the COUNT that times out):

SELECT COUNT(*) FROM (
  (SELECT "p_ci_pipelines".* FROM "p_ci_pipelines"
     WHERE "source" = 10 AND "merge_request_id" = $MR AND "project_id" IN ($SOURCE, $TARGET))
  UNION
  (SELECT "p_ci_pipelines".* FROM "p_ci_pipelines"
     WHERE "project_id" = $SOURCE AND ("source" IN (1,2,3,4,5,6,7,8,11) OR "source" IS NULL)
       AND "ref" = $BRANCH AND "tag" = FALSE AND "sha" = $SHA)
) p_ci_pipelines WHERE "project_id" = $TARGET;

The surviving merge-request-event arm effectively carries project_id IN ($SOURCE, $TARGET) AND project_id = $TARGET, the redundant pair that defeats the planner.

After (single equality-scoped lookup, no UNION):

SELECT COUNT(*) FROM "p_ci_pipelines"
WHERE "source" = 10 AND "merge_request_id" = $MR AND "project_id" = $TARGET;

Database

This is a query change against p_ci_pipelines, so it needs database review (the database label). The plans below were captured on Database Lab (postgres.ai) against gitlab-production-ci at production scale, using the reproduction MR omnibus-gitlab!9563 (closed); on a thin clone the buffers read are the stable metric, while wall-clock time depends on cache state.

Before (target-only COUNT), times out:

Aggregate (actual time=55885.596..55885.615 rows=1)
  Buffers: shared hit=157 read=54565 dirtied=289   [about 426 MiB read]
  ->  Unique  (rows=0)
        ->  Sort   Sort Key: <all 32 columns>        [de-dup over SELECT *, width=1816]
              ->  Append
                    ->  Append
                          ->  Index Scan using index_ci_pipelines_on_project_id_and_status_and_config_source on ci_pipelines (actual time=42244.288..42244.289 rows=0)
                                Index Cond: (project_id = ANY ('{$SOURCE,$TARGET}') AND project_id = $TARGET)
                                Filter: (source = 10 AND merge_request_id = $MR)
                                Rows Removed by Filter: 43674
                                Buffers: shared hit=98 read=41759
                          [partitions 103..117: same project_id Index Scan shape]
                    ->  Result  (One-Time Filter: false)   [branch arm: provably empty, never executed]
Execution time: about 55.9 s (I/O read 55.5 s); 54,565 buffers read.

After (target-only COUNT), equality-scoped:

Aggregate (actual time=2.542..2.556 rows=1)
  Buffers: shared read=50                            [about 400 KiB]
  ->  Append (rows=0)
        ->  Bitmap Heap Scan on ci_pipelines (rows=0)
              Recheck Cond: (merge_request_id = $MR AND project_id = $TARGET AND source = 10)
              Buffers: shared read=4
              ->  BitmapAnd
                    ->  Bitmap Index Scan on index_ci_pipelines_on_merge_request_id   (Index Cond: merge_request_id = $MR)
                    ->  Bitmap Index Scan on index_ci_pipelines_on_project_id_and_source   (never executed)
        [partitions 103..117: same BitmapAnd shape]
Execution time: about 2.6 ms; 50 buffers read.

With the flag disabled the finder runs the "Before" plan above (the timeout), and with it enabled the finder runs the "After" plan, so both flag states are already measured here and no new query measurements are needed.

A source-only viewer keeps the UNION, and each arm is scoped by equality to the source project.

SELECT COUNT(*) FROM (
  (SELECT "p_ci_pipelines".* FROM "p_ci_pipelines"
     WHERE "source" = 10 AND "merge_request_id" = $MR AND "project_id" = $SOURCE)
  UNION
  (SELECT "p_ci_pipelines".* FROM "p_ci_pipelines"
     WHERE "project_id" = $SOURCE AND ("source" IN (1,2,3,4,5,6,7,8,11) OR "source" IS NULL)
       AND "ref" = $BRANCH AND "tag" = FALSE AND "sha" IN ($SHAS))
) p_ci_pipelines;

Source-only COUNT plan:

Aggregate (rows=1)
  Buffers: shared hit=72 read=67                 [139 buffers total, about 1.1 MiB]
  ->  Unique (rows=2)                            [UNION de-duplication]
        ->  Sort   Sort Key: <all 32 columns>
              ->  Append
                    ->  Append   [merge-request-event arm, project_id = $SOURCE]
                          ->  Bitmap Heap Scan on ci_pipelines   [and partitions 103..117]
                                Recheck Cond: (merge_request_id = $MR AND project_id = $SOURCE AND source = 10)
                                ->  BitmapAnd
                                      ->  Bitmap Index Scan on index_ci_pipelines_on_merge_request_id   (merge_request_id = $MR)
                                      ->  Bitmap Index Scan on index_ci_pipelines_on_project_id_and_source   (never executed)
                    ->  Index Scan using index_ci_pipelines_on_project_id_and_sha on ci_pipelines   [branch arm; and partitions 103..117 via project_id + ref/sha indexes]
                          Index Cond: (project_id = $SOURCE AND sha IN ($SHAS))
Total: 139 buffers read; a few milliseconds (wall-clock varies with clone cache state).

The merge-request-event arm uses the selective merge_request_id index via a BitmapAnd, and the branch arm is a set of small project-and-sha index lookups, so the whole source-only path reads 139 buffers, far below the 54,565 of the timing-out path.

Test coverage

spec/finders/ci/pipelines_for_merge_request_finder_spec.rb:

The existing permission contexts (target-only, source-only, both, neither, nil actor) still assert the exact returned pipelines and pass, which guarantees result identity.
The target-only context now asserts the generated SQL scopes by equality ("project_id" = target), contains no "project_id" IN list, and contains no UNION.
The source-only context now asserts the UNION is kept and each arm is scoped by equality to the source project ("project_id" = source), with no "project_id" IN list.
A new context for a non-persisted (unsaved) fork MR with partial permissions asserts the source-only viewer gets the source branch pipeline and the target-only viewer gets nothing.

spec/graphql/resolvers/merge_request_pipelines_resolver_spec.rb:

The existing fork-MR permission contexts pass unchanged.

Closes #629269 (closed)
Reproduction: omnibus-gitlab!9563 (closed) and omnibus-gitlab!9711 (closed)
Related (same statement-timeout family, different endpoint): #365880
Feature flag issue - #634103

Edited Oct 16, 2026 by Shabini Rajadas

Fix fork merge request pipeline count timeout