Evaluate approval policies on merge requests without diff commits

What does this MR do and why?

Approval policies now evaluate merge requests whose latest diff has no commits. Before, their scan_finding and license_scanning violations stayed running forever, because no evaluation and no policy timeout ever found the merge request.

Details

Every pipeline-side evaluation path finds merge requests through Ci::Pipeline#opened_merge_requests_with_head_sha:

  • the findings sync
  • the license sync
  • the scheduling of the policy evaluation timeout
  • the timeout itself

That method starts from all_merge_requests, which joins merge_request_diff_commits. An empty diff has no rows there, so the lookup returns nothing.

For branch pipelines, the method already checks that the latest diff's head_commit_sha matches the pipeline sha or source_sha, so the commits join adds nothing for this caller. With the flag on, branch pipelines start from the source project's merge requests for the pipeline ref instead. Merge request pipelines keep using all_merge_requests, which for them filters on merge_request_id and doesn't join diff commits. all_merge_requests is unchanged.

The result is a superset of the old one. The only merge requests it adds have a latest diff whose head SHA matches without that SHA in its commit rows. In practice those are empty diffs.

  • Behind the gitlab_com_derisk flag policy_head_sha_lookup_without_diff_commits (default off). Rollout issue: #638433
  • Actor: the pipeline's project. The lookup runs before any single merge request is known.
  • No changelog: the flag is off by default.

Reviewable LOC: 11 (plus 100 in tests)

References

  • Resolves #638061
  • Related: #638107 (periodic sweep for running rows left behind by any cause)
  • Related: #617804

Screenshots or screen recordings

Not applicable, backend only.

How to set up and validate locally

  1. Add a merge request approval policy with a scan_finding rule to a project with security scanning in CI.
  2. Push a branch with no commits beyond the default branch and open a merge request from it. Wait for the branch pipeline to finish and for the 150 s timeout.
  3. With the flag off, merge_request.scan_result_policy_violations.pluck(:status) stays running. With Feature.enable(:policy_head_sha_lookup_without_diff_commits), re-run the pipeline: the timeout marks the rows and re-evaluates them.

Specs:

$ bin/rspec ee/spec/models/ci/pipeline_spec.rb:1168
10 examples, 0 failures
$ bin/rspec ee/spec/workers/security/scan_result_policies/unblock_pending_merge_request_violations_worker_spec.rb
40 examples, 0 failures
$ bin/rspec <findings sync, license sync, notification worker specs>
52 examples, 0 failures

Database review

With the flag on, the query for a branch pipeline is:

SELECT merge_requests.* FROM merge_requests
WHERE source_project_id = ? AND source_branch = ? AND state_id = 1
  AND EXISTS (SELECT 1 FROM merge_request_diffs
              WHERE id = merge_requests.latest_merge_request_diff_id
                AND head_commit_sha IN (?, ?))

In the dev DB, the merge request lookup uses idx_merge_requests_on_source_project_and_branch_state_opened, and the diff check is a semi-join on merge_request_diffs. This removes the old union over merge_request_diff_commits and merge_request_commits_metadata.

Plans from a postgres.ai thin clone of the main database, for one open MR on a busy project:

Path Planning Execution Shared buffers
New (flag on) 10 ms 0.11 ms 16 hit
Old (by_commit_sha) 12.6 ms 19.4 ms 26 hit, 12 read

The new path is a nested loop. It does an index scan on idx_merge_requests_on_source_project_and_branch_state_opened, then an index scan on merge_request_diffs_pkey with head_commit_sha as a filter.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Context for LLM agents
  • Rejected: changing Ci::Pipeline#all_merge_requests. It has other callers that rely on the commits match.
  • Rejected: per-caller fixes in the findings sync, license sync and timeout. They all route through this method, so one change covers all four.
  • Flag actor is the pipeline's project, not the merge request's target project. The lookup happens before there is a merge request, and fork merge requests found here can target another project.
  • One existing example (source_sha on a branch pipeline) now also returns the merge request whose head is that source_sha. That setup can't happen in production, because only merged results pipelines set source_sha. The example now has a comment and a flag-off context.

Ci::Pipeline#all_merge_requests has the same gap for head_pipeline_id. It is out of scope here and tracked in #638580.

Edited by Alan (Maciej) Paruszewski

Merge request reports

Loading
Loading