The [Projects List API](https://docs.gitlab.com/ee/api/projects.html#list-all-projects) is showing to perform over our [new performance target of 500ms](https://about.gitlab.com/handbook/engineering/quality/issue-triage/#severity). We (Quality) have recently expanded performance test coverage to include "horizontal" test areas like Projects in a consistent way. As such we now test the API on our 10k Reference Environment against 250 groups with 10 projects each (a total of 2500 projects) at 200 RPS: ``` * Environment: 10k * Environment Version: 13.2.0-pre `744484487f2` * Option: 60s_200rps * Date: 2020-06-30 * Run Time: 1h 4m 21.39s (Start: 01:27:22 UTC, End: 02:31:44 UTC) * GPT Version: v2.0.3 NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT ---------------------------------------------------------|-------|----------------------|------------|-----------------------|-----------------|-------- api_v4_projects | 200/s | 50.68/s (>32.00/s) | 3490.03ms | 4244.82ms (<7500ms) | 100.00% (>95%) | Passed ``` Metrics:  For this endpoint in particular there's some more data to point out. On gitlab.com, data looks to show it performing substantially worse than we see on our smaller environments: >>> Worst performing Rails API routes in the applications, by p95 latency /api/:version/projects p95 latency for this route: 15s (find related issues) >>> The endpoint appears to be then exponential in how it degrades as the number of projects rise. Although .com is indeed an outlier and has a significant number of projects. As part of the [new performance targets](https://about.gitlab.com/handbook/engineering/quality/issue-triage/#severity) this endpoint is currently over our TTFB target and is close enough that it falls just into the ~S2 tier. Although if you want to weigh .com more it could be considered higher. But .com is a live environment with many moving parts so it's not completely reliable so this will be raised as a S3. Task is to reduce the performance of this endpoint consistently to the next tier of ~S3 (<2000ms). **To-do:** https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66598, blocked by: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67936, blocked by: we need to **Introduce a special version of** `EachBatch` that can use composite row comparison. **Doing:** **Done:**