Fix projects dashboard N+1 for latest pipeline

What does this MR do and why?

Uses an updated BatchLoader for Commit#lazy_latest_pipeline to batch on both the project and the SHA.

Part of #214037

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

How to set up and validate locally

1. Change the expected count to 2 in the N+1 test at spec/features/dashboard/projects_spec.rb:254.
2. Run the N+1 test; observe that it passes.
3. Try it with the same threshold master
4. Run the N+1 test; observe that it fails with pipeline and build queries being the offenders

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

Database review

Raw query

Generated locally using the following command, but then adapted the query to high traffic projects on GitLab.com:

projects = Project.all
project_shas = projects.to_h { |project| [project, [project.pipeline_status.sha, '123']] }
Ci::Pipeline.latest_pipeline_per_project_commit(project_shas)
# extract SQL from logs

Query:

SELECT DISTINCT ON (project_id, sha)
    *
FROM
    "ci_pipelines"
WHERE (1 = 0
    OR "ci_pipelines"."project_id" =  278964 -- gitlab-org/gitlab
    AND "ci_pipelines"."sha" IN (
        '90a84065fe9d46bd0344211d202c8ec1771e9109',
        '23d2f6c1a0650677e6cbb6a802f016f85eb7f08e',
        'b4415100354c6d5d6d9f4b6c198ba8fc26b635bb',
        'd2d3817d6ff97e83a6ace3876a40d586418c2312'
    )
    OR "ci_pipelines"."project_id" = 7764 -- gitlab-com/www-gitlab-com
    AND "ci_pipelines"."sha" IN (
        'e70b0eabd2332afa19827c0c334121cae584a672',
        '277030680af7bca6c7a97473eaf76178e6e89f9b',
        '2b139a83c0338f91d8b9b2e5897e8e44d0ca5a36',
        'dbe139011c9594139069022020f6417b5f6fed12' -- no pipeline
    )
    OR "ci_pipelines"."project_id" = 250833 -- gitlab-org/gitlab-runner
    AND "ci_pipelines"."sha" IN (
        '948356fb32b50a5cc149e89c7225a6408110ac46',
        '84b1b297cacafc40bbe5974d30dc082ce08f570c',
        'd3c36e7cd0ae0c8c580e6f7ee8b979c1686439ba',
        '8f4df57f01c1e4f3931bc84907e6e131ed3357a7', -- no pipeline
        '6718b80fac73c6752a89200538b154ca5619e613' -- no pipeline
    )
 )
 ORDER BY
    "ci_pipelines"."project_id" ASC,
    "ci_pipelines"."sha" ASC,
    "ci_pipelines"."id" DESC

Query plan

See also on postgres.ai.

Update: I also generated a larger example. See cold and warm execution links (internal links).

Update 2: There is a 10-condition limit per query based on reviewer feedback and query timings on postgres.ai. This keeps the cold query times manageable. See this cold query plan for an example (internal link).

  Unique  (cost=34.91..34.94 rows=4 width=350) (actual time=61.468..61.483 rows=9 loops=1)
   Buffers: shared hit=69 read=13 dirtied=1
   I/O Timings: read=60.739 write=0.000
   ->  Sort  (cost=34.91..34.92 rows=4 width=350) (actual time=61.466..61.471 rows=11 loops=1)
         Sort Key: ci_pipelines.project_id, ci_pipelines.sha, ci_pipelines.id DESC
         Sort Method: quicksort  Memory: 27kB
         Buffers: shared hit=69 read=13 dirtied=1
         I/O Timings: read=60.739 write=0.000
         ->  Bitmap Heap Scan on public.ci_pipelines  (cost=28.73..34.87 rows=4 width=350) (actual time=51.305..61.382 rows=11 loops=1)
               Buffers: shared hit=63 read=13 dirtied=1
               I/O Timings: read=60.739 write=0.000
               ->  BitmapOr  (cost=28.73..28.73 rows=4 width=0) (actual time=51.284..51.287 rows=0 loops=1)
                     Buffers: shared hit=55 read=10
                     I/O Timings: read=50.926 write=0.000
                     ->  Bitmap Index Scan using index_ci_pipelines_on_project_id_and_sha  (cost=0.00..8.84 rows=2 width=0) (actual time=51.143..51.144 rows=3 loops=1)
                           Index Cond: ((ci_pipelines.project_id = 278964) AND ((ci_pipelines.sha)::text = ANY ('{90a84065fe9d46bd0344211d202c8ec1771e9109,23d2f6c1a0650677e6cbb6a802f016f85eb7f08e,b4415100354c6d5d6d9f4b6c198ba8fc26b635bb,d2d3817d6ff97e83a6ace3876a40d586418c2312}'::text[])))
                           Buffers: shared hit=10 read=10
                           I/O Timings: read=50.926 write=0.000
                     ->  Bitmap Index Scan using index_ci_pipelines_on_project_id_and_sha  (cost=0.00..8.84 rows=1 width=0) (actual time=0.072..0.072 rows=5 loops=1)
                           Index Cond: ((ci_pipelines.project_id = 7764) AND ((ci_pipelines.sha)::text = ANY ('{e70b0eabd2332afa19827c0c334121cae584a672,277030680af7bca6c7a97473eaf76178e6e89f9b,2b139a83c0338f91d8b9b2e5897e8e44d0ca5a36,dbe139011c9594139069022020f6417b5f6fed12}'::text[])))
                           Buffers: shared hit=20
                           I/O Timings: read=0.000 write=0.000
                     ->  Bitmap Index Scan using index_ci_pipelines_on_project_id_and_sha  (cost=0.00..11.05 rows=1 width=0) (actual time=0.065..0.065 rows=3 loops=1)
                           Index Cond: ((ci_pipelines.project_id = 250833) AND ((ci_pipelines.sha)::text = ANY ('{948356fb32b50a5cc149e89c7225a6408110ac46,84b1b297cacafc40bbe5974d30dc082ce08f570c,d3c36e7cd0ae0c8c580e6f7ee8b979c1686439ba,8f4df57f01c1e4f3931bc84907e6e131ed3357a7,6718b80fac73c6752a89200538b154ca5619e613}'::text[])))
                           Buffers: shared hit=25
                           I/O Timings: read=0.000 write=0.000

Statistics:

Time: 62.651 ms
  - planning: 0.942 ms
  - execution: 61.709 ms
    - I/O read: 60.739 ms
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 69 (~552.00 KiB) from the buffer pool
  - reads: 13 (~104.00 KiB) from the OS file cache, including disk I/O
  - dirtied: 1 (~8.00 KiB)
  - writes: 0