Avoid sha-filtered pipeline lookup that can time out
What does this MR do and why?
Project#latest_pipeline filtered on the HEAD sha, which can match a huge number of pipelines on one commit and time out. Behind a feature flag, fetch via the (project_id, ref, id DESC) index instead by getting the last pipeline and checking whether that sha matches what we need.
Context about related issue
The securityScanners GraphQL field resolves the project's latest default-branch pipeline via Project#latest_pipeline, which runs ... WHERE project_id AND ref AND sha ORDER BY id DESC LIMIT 1. A commit sha is normally selective, but for the project mentioned in the issue there are thousands of pipelines. This causes a timeout and produces a 503 on the GraphQL query.
The fix fetches the newest pipeline for the ref using the (project_id, ref, id DESC) index and returns it when it already matches the requested sha (the common case). It only falls back to the original sha-filtered lookup when the newest pipeline is a different sha, so results are unchanged. Gated behind the latest_pipeline_ref_first_lookup feature flag (gitlab_com_derisk, default off).
References
Related #600131
Database
Both queries validated in Database Lab against the affected project (CI database).
Before: sha-filtered lookup (project_id, ref, sha ORDER BY id DESC LIMIT 1) https://console.postgres.ai/gitlab/projects/gitlab-production-ci/sessions/53495/commands/155549
SELECT p_ci_pipelines.* FROM p_ci_pipelines
WHERE project_id = <project_id>
AND (source IN (1,2,3,4,5,6,7,8,10,11) OR source IS NULL)
AND ref = 'master' AND sha = '<HEAD_sha>'
ORDER BY id DESC LIMIT 1;Uses index_ci_pipelines_on_project_id_and_sha, (~88k matching rows). ~54 s, ~165k shared buffers (~1.2 GiB).
After: ref-first lookup (project_id, ref ORDER BY id DESC LIMIT 1) https://console.postgres.ai/gitlab/projects/gitlab-production-ci/sessions/53495/commands/155550
SELECT p_ci_pipelines.* FROM p_ci_pipelines
WHERE project_id = <project_id>
AND (source IN (1,2,3,4,5,6,7,8,10,11) OR source IS NULL)
AND ref = 'master'
ORDER BY id DESC LIMIT 1;Uses index_ci_pipelines_on_project_idandrefandiddesc, returns row 1 immediately. ~62 ms, 60 shared buffers.
For the affected project it was verified that the newest pipeline on the ref is HEAD's sha, so the fast path is taken (no fallback).
How to set up and validate locally
-
Enable the feature flag:
Feature.enable(:latest_pipeline_ref_first_lookup) -
For a project with pipelines on its default branch, query the
securityScannersfield in the GraphQL explorer (/-/graphql-explorer):query { project(fullPath: "<path>") { securityScanners { available enabled pipelineRun } } } -
Confirm it returns the scanner data. With the flag disabled the behavior is unchanged (falls back to the previous lookup).