Investigation of why `GET /api/:version/groups/:id/merge_requests` is slow and times out, run on 2026-09-30: a static code trace, GDK2 tracing and timing, GitLab.com black-box timing and prior-art research. This issue records the method, the numbers, the ranked fixes and where the scripts, dataset and raw data live, so the work can be picked up cold. - On GitLab.com the API default `state=all` returns 408 after 15-20 s for gitlab-org and gitlab-com whatever else is passed; only requests that shrink the candidate set first succeed (`state=opened`: 4.2 s). No index serves `ORDER BY created_at DESC, id DESC` across the projects join, so Postgres sorts the whole filtered set and the COUNT is a second pass over it. - Once the SQL fits, the request is Ruby/Gitaly/Redis bound, not database bound: on GDK2 `detailed_merge_status` alone accounts for 537 of the 693 ms serialize baseline for a page of 100 open MRs. Earlier attempts (!231260, !225446, !226653) were rolled back or disabled, and the July retro (#606010) concluded none worked. - Fix 1: make `detailed_merge_status` cheap in list context (skip the per-MR Gitaly conflict check, cut and batch the Redis reads, drop the per-MR log line); expected serialize phase 537 -> ~150-200 ms per 100 open MRs locally. - Fix 2: fix the preload chain, starting with dropping `.preload(:source_project, :target_project)` in `find_merge_requests` (-28 to -34 route queries per page); small, no contract change. - Fix 3: give `state=all` on large groups an index-driven access path, the only thing that turns the 408s into 200s; the next experiment is an opt-in keyset path built on the IN-operator optimisation from !231260, after asking why the May trial was abandoned. - Fix 4: fix #602884 (`view=simple` enqueues `MergeabilityCheckBatchWorker` for every page) and pair it with #628132 (no `mark_as_checking` writes on GET); one-line fix plus spec. <details> <summary>Detailed context for AI agents</summary> ### Scope and method Run on 2026-09-30 by Marc Shaw (marc_shaw): four parallel investigations (static code trace, GDK2 dynamic tracing/timing, GitLab.com black-box timing, prior-art research) plus manual verification. Repo HEAD 809f76cdb113 (origin/master 2026-09-30). - **Static code trace (read-only).** Full call graph from the Grape route through `MergeRequestsFinder`, `IssuableFinder::Params#projects`, `OffsetPagination`, `with_api_entity_associations` (CE + EE), every `expose` in `Entities::MergeRequestBasic` (+ EE prepend), `DetailedMergeStatusService` -> `RunChecksService` -> 19 check classes, the response cache, comparison with the group issues API and GraphQL. Finder SQL captured for 30 parameter variants (`static/to_sql.out`). Write-up: `static/trace.md` (70 KB). - **GDK2 dynamic tracing and timing.** - Dataset: see "GDK2 dataset" below. - Harness: in-process full Rack stack requests (`trace.rb`) with a `sql.active_record` subscriber (normalised SQL, counts, ms, `Gitlab::BacktraceCleaner` callers), taps on `Gitlab::GitalyClient.execute` and the low-level `RedisClient#call*`, warm-up + median of 3; `phases.rb` (auth / finder / count / select / preload / serialize / json), `exposures.rb` (disable one exposure at a time via in-memory prepend), `callers.rb`, `profile.rb` (StackProf), `plans.rb` (EXPLAIN ANALYZE BUFFERS), `curl_bench.sh` (HTTP via puma + api_json.log fields), `bullet_ab.rb`. - 44 scenarios (see "GDK2 scenario list" below), scaling at per_page 20/50/100, per-exposure attribution for open and merged pages. - Complex EE structures via throwaway request specs with factories and QueryRecorder in `/Users/marcshaw/work/gdk2/gitlab/tmp/mrapi_specs/` (approval rules, head pipelines, unresolved discussions, blocking MRs, external status checks, policy reads/violations, requested changes, title regex, ff/rebase merge methods with real branches, path and LFS locks). - **GitLab.com black-box timing (read-only GETs, 2026-09-30 20:15-21:00 UTC).** 83 requests total (81 REST + 2 GraphQL) with Marc's token, curl timings, headers saved (`prod/raw/`), X-Request-Ids recorded (`prod/request_ids.txt`, KQL in `prod/request_ids_kql.txt`). Groups: gitlab-org (9970), gitlab-com (6543), gitlab-org/quality (2750817), gitlab-org/incubation-engineering (12892338). Kibana needs SSO (401 via curl), so server-side percentiles were not measured; ready-to-open Discover links are in `prod/kibana_urls.md`. - **Prior art.** git history of the endpoint files, gitlab.com issues/MRs/epics, feature-flag-log, comparison with `GET /groups/:id/issues`, GraphQL `group.mergeRequests`, the Vue group MR list, indexes on `merge_requests`. Catalog with links: `priorart/catalog.md`. ### Findings in detail - **GitLab.com.** On GitLab.com the endpoint is unusable for the two largest groups with the API default `state=all`: every request to gitlab-org (9970) and gitlab-com (6543) returned 408 `Request timed out` (15-20 s), whatever else was passed (per_page=20, any page, any order_by, view=simple, labels, search, non_archived=false). Only requests that shrink the candidate set first succeed (state=opened 4.2 s for 4,855 MRs; assignee/reviewer/author/scope filters 2-4 s; milestone 4.4 s). The cause is the query shape: no index serves `ORDER BY created_at DESC, id DESC` across the projects join, so Postgres sorts the whole filtered set (millions of rows for gitlab-org), and the COUNT (LIMIT 10001) is a second pass over the same set. The 408 durations cluster at 15.3-15.7 s (one statement cancelled) and 17.3-20.1 s (count finished in 2-5 s, then the sorted SELECT was cancelled). - **Once the SQL fits (`state=opened`).** The request is Ruby/Gitaly/Redis bound, not database bound. GDK2, page of 100 open mergeable MRs, Bullet off: 641 ms total, of which entity serialization 537 ms; `detailed_merge_status` alone accounts for 537 of the 693 ms serialize baseline (all 46 Gitaly calls, ~1,000 of the 1,136 Redis commands, 67 of 108 SQL). Per open MR: 1 Gitaly `FindCommit` (conflict check), ~11 Redis commands, 17 check classes, 1 JSON log line. Production: gitlab-org state=opened 4.2 s wall with 2.4 s DB (of which COUNT 1.24 s, measured in !257173), so ~1.8 s is this per-record work. ### GitLab.com timings per_page=100 unless noted, r1/r2 back to back; TTFB equals total within 0.4 s. | scenario | group | status r1/r2 | total s r1/r2 | body bytes | X-Total | |---|---|---|---|---|---| | default (state=all) | gitlab-org | 408/408 | 18.99/19.61 | 41 | - | | per_page=20 | gitlab-org | 408/408 | 17.52/19.61 | 41 | - | | page=2 / 50 / 100 | gitlab-org | 408 x6 | 15.4-18.4 | 41 | - | | state=opened | gitlab-org | 200/200 | 4.20/4.18 | 607,831 | 4855 | | state=merged | gitlab-org | 408/408 | 15.59/15.74 | 41 | - | | state=closed | gitlab-org | 408/408 | 17.67/15.48 | 41 | - | | scope=all | gitlab-org | 408/408 | 15.45/15.36 | 41 | - | | scope=created_by_me | gitlab-org | 200/200 | 3.21/2.12 | 1,083,573 | 1048 | | scope=assigned_to_me | gitlab-org | 200/200 | 3.21/3.52 | 1,107,624 | 988 | | scope=reviews_for_me | gitlab-org | 200/200 | 7.85/3.92 | 793,131 | 820 | | labels=backend | gitlab-org | 408/408 | 15.43/15.42 | 41 | - | | labels=backend&state=opened | gitlab-org | 200/200 | 3.15/3.74 | 684,120 | 1777 | | not[labels]=backend | gitlab-org | 408/408 | 19.65/17.31 | 41 | - | | labels=Category:Code Review Workflow (state=all) | gitlab-org | 200/200 | 0.95/1.24 | 102,941 | 10 | | milestone=19.5 (state=all) | gitlab-org | 200/200 | 4.43/4.51 | 642,717 | 7295 | | milestone=19.5&state=opened | gitlab-org | 200 | 4.51 | 646,432 | 1767 | | assignee_username=marc_shaw | gitlab-org | 200/200 | 2.01/2.42 | 1,107,624 | 988 | | reviewer_username=marc_shaw | gitlab-org | 200/200 | 2.88/3.19 | 793,131 | 820 | | author_username=marc_shaw | gitlab-org | 200/200 | 3.27/3.53 | 1,083,573 | 1048 | | search=timeout (also &in=title) | gitlab-org | 408 x4 | 15.3-15.6 | 41 | - | | search=timeout&state=opened | gitlab-org | 200 | 6.77 | 1,410,109 | 270 | | order_by=updated_at / title / milestone_due / priority / label_priority | gitlab-org | 408 x10 | 15.3-19.3 | 41 | - | | order_by=title&state=opened | gitlab-org | 200 | 5.78 | 1,269,059 | 4851 | | created_after=30d | gitlab-org | 200/200 | 5.00/5.98 | 611,540 | absent (>10k) | | updated_after=7d | gitlab-org | 200/200 | 3.90/4.31 | 609,629 | 9014 | | merged_after=7d&state=merged | gitlab-org | 200/200 | 8.58/9.89 | 523,742 | 3466 | | target_branch=master&state=opened | gitlab-org | 200/200 | 3.17/2.29 | 638,325 | 2207 | | draft=true&state=opened | gitlab-org | 200/200 | 2.73/2.44 | 615,852 | 1319 | | with_labels_details=true | gitlab-org | 408/408 | 15.41/15.42 | 41 | - | | view=simple | gitlab-org | 408/408 | 15.44/15.63 | 41 | - | | non_archived=false | gitlab-org | 408/408 | 18.87/15.59 | 41 | - | | with_merge_status_recheck=true&per_page=20&state=opened (once) | gitlab-org | 200 | 1.83 | 107,919 | 4851 | | page=49&state=opened (last page, 51 items) | gitlab-org | 200 | 3.00 | 900,536 | 4851 | | default | gitlab-com | 408/408 | 20.14/15.48 | 41 | - | | default | gitlab-org/quality | 200/200 | 8.16/7.56 | 665,101 | absent (>10k) | | default | gitlab-org/incubation-engineering | 200/200 | 3.80/2.25 | 292,240 | 328 | | GraphQL group.mergeRequests(first:100, state:opened, sort:CREATED_DESC) | gitlab-org | 200/200 | 3.98/6.31 | 56,147 | keyset, no count | Observations: - 42 of 81 REST requests were 408; 200s ranged 0.95-9.89 s (median 3.5 s). No 5xx, no 429. - Offset is irrelevant (page 49 of state=opened is 3.0 s). - X-Total absence does not correlate with time; the candidate set size before ORDER BY does. - Cold vs warm is flat. - Payload is 5-14 KB per MR uncompressed, transfer 0.05-0.36 s, so the server is the cost. - 41 of 100 open gitlab-org MRs came back with `detailed_merge_status: unchecked`. - GraphQL is in the same latency band as REST state=opened despite an 11x smaller payload because it resolves `detailedMergeStatus` per node too. ### Production numbers quoted from issues (not re-measured) - weekly-status-check-in#110 (2025-06-22..28): 407,055 apdex failures of 3,251,245 requests (12.5% over the 5 s target), number 2 Code Review error-budget contributor. - #552376 (2025-07): p95 of the two list endpoints 14-16 s, top 5 root namespaces = 40.8% of slow requests, almost all polling bots. - #558535 (2025-08): 2,830,011 requests, 201,164 over 5 s. - #595654 (2026-04): scope=all + search automation 100% 408, "majority of DB timeouts come from the COUNT query". - triage-support-fix#10 (2025-04-11) api_json sample of a 408 on gitlab-org labels=...&state=merged: duration_s 15.196, db_duration_s 15.103, cpu_s 0.16. - !257173 (2026-09-23): COUNT = 1,240 of 2,400 ms DB time on gitlab-org; flamegraphs of `state=opened&per_page=100&page=18`: 8.8 s wall / 1.9 s count, 6.4 s / 2.25 s. - !231260 analysis: 86% of >= 5 s requests carried only `[page, per_page, state, with_merge_status_recheck]`. ### GDK2 dataset (left in place) - Group `mrapi-perf-bf9eb4` (id 4157), 3 subgroups, 30 regular projects (public/internal/private round robin, merge methods merge/ff/rebase_merge, half with `only_allow_merge_if_all_discussions_are_resolved`, a third with `only_allow_merge_if_pipeline_succeeds`, 15 with a project approval rule) + 2 archived + 1 private project with MRs disabled. - 6,000 MRs: 1,922 opened / 3,532 merged / 546 closed, 885 drafts, 312 fork-style, 167 with real branches created through `MergeRequests::CreateService`, the rest bulk-inserted; metrics and diffs for all. - Other data: 12,143 label links (15 group + 165 project labels), 6,082 assignees, 9,190 reviewers with mixed states, 49% with milestones, 31,452 notes (672 unresolved DiscussionNotes on 351 open MRs), 3,500 award emoji, 888 timelogs, 3,207 approvals, 726 head pipelines in the ci database. - Actors: `mrapi-dev` (id 3730, non-admin Developer of the root group), `mrapi-outsider` (3731, non-member), root. X-Total 5,814 (dev) / 3,883 (outsider). - Details: `dynamic/dataset.md`; seed script `dynamic/seed.rb`. ### GDK2 phase breakdown and scaling Bullet off, in-process, page of 100, median of 3. | phase | default (100 open, can_be_merged) | state=merged | |---|---|---| | auth + group lookup | 5 ms, 6 SQL | 5 ms, 6 SQL | | finder build | 2 ms | 2 ms | | COUNT (LIMIT 10001) | 3.6 ms, 1 SQL | 3.1 ms | | main SELECT | 10 ms, 2 SQL | 10 ms | | `with_api_entity_associations` preload | 78 ms, 37 SQL | 64 ms, 31 SQL | | entity serialization | 537 ms, 100 SQL, 46 Gitaly (153 ms), ~1,100 Redis | 194 ms, 56 SQL, 0 Gitaly, 9 Redis | | JSON encode | 6 ms (299 KB) | 7 ms (364 KB) | | total | 641 ms wall / 419 ms cpu, 147 SQL (38 ms) | 286 ms / 258 ms, 98 SQL (26 ms) | - Full HTTP request through puma (Bullet on, so ~3x): default 3.51 s / cpu 3.26 s / db_count 299 (142 cached) / gitaly 46 (219 ms); view=simple 158 ms. - Bullet off in-process: default 695 ms, state=merged 370 ms, view=simple 60 ms (23 SQL). - Scaling (least squares over per_page 20/50/100, default page): wall 6.7 ms per MR + 18 ms fixed; cpu 4.5 ms per MR; SQL 0.81 per MR + 79 fixed (the "per MR" SQL is really per distinct project); Redis 11.3 commands per MR; Gitaly 0.45 per MR (0.52 per non-draft open mergeable MR); 2.94 KB per MR. ### GDK2 scenario list Bullet off, actor `mrapi-dev`, per_page=100 unless noted; wall ms / SQL uncached / Redis cmds / Gitaly. `default` 695 / 157 / 1136 / 46; `per_page=20` 163 / 90 / 233 / 10; `per_page=50` 337 / 128 / 587 / 24; `page=2` 488 / 169 / 732 / 52; `page=30` 383 / 151 / 93 / 0; `state=opened` 658 / 157 / 1136 / 46; `state=merged` 370 / 138 / 9 / 0; `scope=all` 654 (= default); `scope=assigned_to_me` 438 / 169 / 303 / 5; `scope=reviews_for_me` 483 / 163 / 320 / 9; `labels=<one>` 489 / 177 / 398 / 14; `labels=<two>` (63 results) 323 / 146 / 91 / 0; `not[labels]` 712 / 167 / 1099 / 48; `labels=None` 422 / 168 / 309 / 10; `milestone` 403 / 152 / 282 / 4; `assignee_username` 449 / 161 / 194 / 3; `assignee_id=None` 461 / 169 / 530 / 19; `reviewer_username` 416 / 151 / 109 / 0; `author_username` 434 / 164 / 235 / 2; `search=refactor` 503 / 166 / 185 / 3 (same with `in=title`); `order_by=updated_at` 684 / 158 / 1113 / 46; `order_by=title` 444 / 160 / 163 / 0; `order_by=milestone_due` 396 / 107 / 91 / 0; `order_by=priority` 403 / 107 / 91 / 0; `order_by=label_priority` 357 / 64 / 133 / 0; `created_after+before` window 380 / 146 / 81 / 0; `updated_after=1mo` 710 / 157 / 1139 / 46; `merged_after=3mo` 407 / 138 / 9 / 0; `target_branch=main` 678 / 157 / 1136 / 46; `source_branch` (30 results) 417 / 172 / 485 / 16; `draft=true` 381 / 135 / 9 / 0; `my_reaction_emoji=thumbsup` 442 / 164 / 229 / 6; `approved_by_ids[]=<id>` (89 results) 391 / 146 / 117 / 0 (scalar `approved_by_ids=<id>` returns HTTP 400); `with_labels_details=true` 768 / 178 / 1136 / 46 (+25 KB); `view=simple` 60 / 23 / 3 / 0; `with_merge_status_recheck=true` 653 / 157 / 1138 / 46 (enqueues one `MergeabilityCheckBatchWorker` job with all 100 ids); `non_archived=false` 728 / 157 / 1140 / 46 (X-Total 5,938); outsider `default` 539 / 155 / 1065 / 58 (X-Total 3,883); outsider `state=opened` 519 / 155 / 1065 / 58; root `default` 639 / 154 / 1136 / 46. The wall time tracks the number of open, non-draft, `can_be_merged` MRs on the page (page 1 of the default sort happens to be 100 such MRs from 24 projects, the worst case); pages of merged/closed/draft MRs cost ~3.7 ms per MR because `CheckOpenStatusService`/`CheckDraftStatusService` fail fast. ### Per-record cost catalogue Default page of 100 open MRs, attributed to callers. 1. `MergeRequests::Mergeability::DetailedMergeStatusService#execute` per MR (`lib/api/entities/merge_request_basic.rb:139`): 537 ms per 100 MRs (disabling it drops the serialize phase from 693 to 156 ms, removes 67 SQL, all 46 Gitaly calls and ~1,000 Redis commands). 17 check classes per MR through `RunChecksService` (execute_all false, stops at first failure), each wrapped in `Mergeability::Logger#instrument` (`clock_gettime` 7% of CPU samples) and one `application_json.log` JSON line per MR (100 per page; `IO#write_nonblock` 13% of CPU samples). Items 2-6 are inside it. 2. Gitaly `FindCommit` per open non-draft `can_be_merged` MR that reaches `CheckConflictStatusService` (`app/services/merge_requests/mergeability/check_conflict_status_service.rb:12`): `merge_request.source_branch_sha != merge_request_diff&.head_commit_sha` -> `MergeRequest#source_branch_head` (`app/models/merge_request.rb:1352`) -> `Repository#commit` -> Gitaly. 46 calls / 153 ms locally (3.3 ms each). Request-store keyed by (repo, ref), so never shared across MRs; the check is not cacheable. ff/rebase_merge projects add a second `FindCommit` for `target_branch_sha` (`CheckRebaseStatusService`); `Repository#ancestor?` is Redis-cached after the first call. In a spec this page trips the Gitaly N+1 detector (46 > 30); the runner needed `GITALY_DISABLE_REQUEST_LIMITS=1`. 3. Redis, 11.4 commands per MR (1,136 per page), none batched: - 3 `GET merge_request:<id>:{approval_check,lfs_file_locks_mergeability,path_locks_mergeability}` per non-draft open MR (`Gitlab::MergeRequests::Mergeability::RedisInterface#retrieve_check`, `lib/gitlab/merge_requests/mergeability/redis_interface.rb:17`), for the three cacheable checks (CheckLfsFileLocksService, EE CheckPathLocksService, EE CheckApprovedService with a 60 s TTL). The LFS/path lock GETs happen even when LFS/locks are inactive for the project. - 2 `GET project:<id>:{lfs_file_locks,path_locks}_changed_epoch` per MR (`Project#get_epoch_from`, `app/models/project.rb:3926`) just to build the cache keys above. - 1.3 `EXISTS mr_<id>_cannot_merge` per MR (`ApprovalState#legacy_temporarily_unapproved?`, `ee/app/models/approval_state.rb:329`), evaluated in `CheckApprovedService#cacheable?` and again in `DetailedMergeStatusService#execute`. - 4 commands (MULTI, SISMEMBER, EXISTS, EXEC on `branch_names:<project>:set`) per MR from `Repository#branch_names_include?` via `Gitlab::SetCache#try_include?` (`lib/gitlab/set_cache.rb:64`), called by `MergeRequest#branch_missing?` in `CheckCommitsStatusService` (`ref_existence_check_gitaly` OFF path; with the flag ON it becomes one `RefService#ListRefs` Gitaly call per distinct project via BatchLoader). - 0.34 `HVALS pipeline_creation:...` per MR (`Ci::PipelineCreation::Requests.for_merge_request`, `app/models/ci/pipeline_creation/requests.rb:91`) for MRs in projects with `only_allow_merge_if_pipeline_succeeds`; `pipeline_creating?` is not memoised and can run up to 3x per MR. - 3 `GET cache:gitlab:{exists?,has_visible_content?,root_ref}:<project>` per distinct project. 4. SQL per distinct project on the page (24-35 of 30 projects per page here; per MR in a group where every MR is from a different project): - `routes` 28 (open page) / 34 (merged page): `Routable#full_path` <- `Project#repository` (via `MergeRequest#target_branch_exists?` in `CheckCommitsStatusService`) and `IssuableReferences#to_reference`. Root cause: `find_merge_requests` (`lib/api/merge_requests.rb:109-110`) does `paginate(...).preload(:source_project, :target_project)` then `.with_api_entity_associations`; the plain `:target_project` preload combined with the nested `target_project: [:route, { namespace: :route }]` from `preload_routables` leaves `target_project.route` unloaded (verified: `association(:route).loaded?` false for all 20 MRs with the plain preload, true without it; `source_project.route` is loaded in both cases; script `verify/route_preload.rb`). - `project_ci_cd_settings` 24 / 30: `EE::Project#merge_trains_enabled?` <- `MergeTrains::Train.project_using_ff?` <- `EE::MergeRequests::Mergeability::DetailedMergeStatusService#skip_rebase_check?` (`ee/app/services/ee/merge_requests/mergeability/detailed_merge_status_service.rb:11-18`), part of `check_params`, so it runs for merged/closed MRs too; `ci_cd_settings` is not preloaded. - `approval_project_rules` 24 / 13: `EE::MergeRequest#missing_policy_approval_rules?` (`ee/app/models/ee/merge_request.rb:517`) <- `CheckSecurityPolicyViolationsService#approval_rules_out_of_sync?`, per MR, deduplicated per project only by the AR query cache. - `security_orchestration_policy_configurations` EXISTS 12 + `namespace_settings`/parent `namespaces` PK 3-4: `Project#has_linked_configurations?` <- cascading `only_allow_merge_if_pipeline_succeeds?` (`ee/app/models/ee/project.rb:683-728`, `CascadingNamespaceSettingAttribute`) read in `CheckCiStatusService` and, for `blocking_discussions_resolved`, `only_allow_merge_if_all_discussions_are_resolved?`. - `projects` PK 4 / 10: `Gitlab::UrlBuilder.build(milestone)` -> `milestone.project` (`config/routes/directs/milestone.rb:9`); `milestone` is preloaded, its project/group and their routes are not (the Issue API preloads `milestone: { project: [:route, { namespace: :route }] }`, `app/models/issue.rb:241-247`). - `merge_trains` 1 per MR in ff/rebase projects (`EE::MergeRequest#should_be_rebased?` -> `merge_train_car`). - `status_check_responses` 1 per MR when external status checks are enforced (spec probe B). - `head_pipeline` (`ci_pipelines`, CI database) 1 per MR when the CI check is reachable (`only_allow_merge_if_pipeline_succeeds` or auto-merge), via `partitionable_belongs_to_loader`; not preloaded. 5. Serialization CPU on a merged page (low noise): user sub-entities (author, assignees, reviewers, merged_by, merge_user, closed_by) 47 ms per 100, `time_stats` 42 ms, `milestone` 21 ms (+14 SQL), `detailed_merge_status` 20 ms + 30 SQL (the `skip_rebase_check?` lookups), `task_completion_status` 15 ms, `references` 14 ms (+30 SQL routes), `issuable_metadata` 11 ms (2 grouped queries), residual 27 ms. 6. Preload phase: 37 statements, 78 ms per page (one per association branch; heavy rows: `unresolved_notes` loads full Note rows incl. `note`, `note_html`, `position` although only type/system/resolved_at are read and `mergeable_discussions_state?` short-circuits when the project does not require resolved threads; EE `blocking_merge_requests` loads full MergeRequest rows). Preloads not consumed on this route: `namespace_settings_with_ancestors_inherited_settings`, `target_project.project_feature`, EE `approved_by_users`, the `applicable_post_merge_approval_rules` tree (only used for merged MRs in ApprovalState, which never runs because `CheckOpenStatusService` fails first), `group: :saml_provider`, `source_project: { namespace: :route }`. 7. GC = 30% of CPU samples of the profiled request. 8. COUNT is cheap locally (3-4 ms vs 7 ms select) but is a second full pass over the filtered set; on gitlab-org it is 1.2-2.25 s (!257173). ### Confirmed bugs - (a) `find_merge_requests` adds `.preload(:source_project, :target_project)` before `with_api_entity_associations`, which makes Rails drop the nested `target_project: [:route, ...]` preload; `target_project.route` is loaded lazily per distinct project (28-34 route queries per page). Verified in a runner: with the plain preload `target_project.association(:route).loaded? == false` for all 20 MRs, without it `true`. - (b) `view=simple` never sets `skip_merge_status_recheck`, so every simple-view page enqueues `MergeabilityCheckBatchWorker` for all MRs on the page (#602884). - (c) EE `DetailedMergeStatusService#skip_rebase_check?` runs `MergeTrains::Train.project_using_ff?` -> `project.merge_trains_enabled?` -> `ci_cd_settings` (not preloaded) for every MR including merged/closed ones (24-30 queries per page). ### Per-exposure attribution caveat On the open page only the `detailed_merge_status` knob is trustworthy (693 -> 156 ms, 67 SQL, 46 Gitaly). The other single-knob deltas (110-209 ms each) sum to far more than the baseline and are measurement noise; treat them as unknown. The merged-page numbers in catalogue item 5 are consistent and usable. ### Complex EE structures (spec probes) Test env, per-MR deltas. | structure | SQL / MR | Redis / MR | Gitaly / MR | note | |---|---|---|---|---| | plain open can_be_merged MRs, real branches | 0.0 | 5.6 | 0.6 FindCommit + 1 ListRefs per project | `ref_existence_check_gitaly` ON in tests | | + approval rule (1 required) + head pipeline + unresolved discussion + discussions-resolved + pipeline-must-succeed | 0.2 (ci db pipeline lookups) | 3.8 | 0.2 | fails fast at discussions_not_resolved / ci_still_running | | blocking MRs + external status check + policy read/violation + requested changes + title regex | 1.8 (`status_check_responses` per MR) | 0.0 | 0.0 | all stop at `status_checks_must_pass` (trivial tier) | | merge_method ff / rebase_merge | 0.6 (`merge_trains`) | 6.2 | 0.6 (2 FindCommit per mergeable MR) | `ancestor?` Redis-cached, no IsAncestor RPC in steady state | | path lock + LFS lock on changed file | 0.0 | 5.6 | 0.6 | cacheable checks served from ResultsStore in steady state | ### Query shape Local plans on 6k rows; shapes only. - Main SELECT: `Limit <- Sort (top-N heapsort on created_at DESC, id DESC)` over the whole filtered join (5,814 rows) <- `Nested Loop Anti Join` (namespace_settings archived EXISTS per row) <- `Hash Anti Join` (banned_users, `author_id + 0`) <- `Nested Loop` over the project list (from the `namespace_descendants` cache row via UNNEST, else `traversal_ids @>`), project_features and namespaces hash-joined, `project_authorizations` EXISTS for private projects <- `merge_requests` by target_project_id. - COUNT = same plan without the sort, `LIMIT 10001`. - No index serves `created_at DESC, id DESC` across the projects join: production either walks `index_merge_requests_on_created_at` backwards with join filtering (bad for small groups) or scans per project and sorts (bad for big groups). - Indexes available are per project only: `(target_project_id, created_at, id)`, `(target_project_id, updated_at, id)`, `(target_project_id, state_id, created_at, id)`; no `(target_project_id, state_id, updated_at, id)`, no trigram on title, no namespace/traversal column on merge_requests (issues have `namespace_id`). - `order_by=title` sorts on `lower(title)`; `milestone_due` adds a hash join and 4-key sort; `priority`/`label_priority`/`popularity` add GROUP BY over the whole filtered set (4x default locally). - `search` is a MATERIALIZED CTE fence then ILIKE over the whole CTE (MergeRequest has no pg_full_text_searchable columns). - `scope=assigned_to_me` is the only index-driven shape (starts from `merge_request_assignees.user_id`). - Full finder SQL for 30 variants: `static/to_sql.out`; plans: `dynamic/plans/*.txt`. ### Measurement caveats for anyone re-measuring on GDK - Bullet (dev-only N+1 detector, active in GDK puma and in the Rack stack) costs ~3x on this endpoint (2,281 ms vs 726 ms interleaved A/B). Disable it for timings; counts are identical. - `Gitlab::Instrumentation::Redis` is disabled in development (`config/initializers/7_redis.rb`), so `api_json.log` has no `redis_*` fields locally and request-store counters read 0. Redis counts came from a low-level `RedisClient` tap; Redis durations were not measurable locally. - `GITALY_DISABLE_REQUEST_LIMITS=1` is needed in runners (46-58 Gitaly calls per page exceed the 30-call limit). - `mem_bytes` is absent from GDK's api_json.log. Test env has all feature flags on (e.g. `ref_existence_check_gitaly`), GDK2 has them off. ### Prior art timeline (do not redo) The team already tried the IN-operator optimisation (!231260, flag `group_mr_in_operator_optimization`, trialled per group for two days in May 2026, removed in !244663 without a written result), a traversal_ids JOIN (!225446, rolled back the same day it reached 100%), and response caching (!226653, disabled after Redis incident gitlab-com/gl-infra/production#22302, now bot-only and off). The July retro (#606010) concluded none of the attempts worked and moved towards recalibrating the SLO (&23233). - 2023-05/07 #417738: 50k reference architecture runs fail the endpoint's 1000 rps target (TTFB avg 2.8-18.7 s). - 2025-06/08 #552376 (Patrick, epic &17943): shipped !196128 (approval-rule preloads, ~900 -> <100 queries, 1.7 -> 1.3 s) and !197105 (`banned_users.user_id = author_id + 0`, next-page query 34 s -> 27 ms on postgres.ai, ~600 -> ~100 ms in prod). Rate limiting (100 rpm/user) and keyset pagination were raised and dropped. - 2025-12 to 2026-07 epic &20250 (closed) + spike #585222 (closed): base gitlab-org query times out on the `namespace_descendants` UNNEST + authorization EXISTS; DB-lab traversal_ids JOIN cut buffers ~17 GB -> ~4 GB but "still prone to timeout"; only known real fix is denormalising traversal ids (rejected for the 342 GB table) or Elasticsearch; CPU dominates DB for slow requests; the one heavy per-row operation is `detailed_merge_status` -> approval checks -> committers. Wrap-up 2026-06-15: "tried most quick wins, decided against any intensive DB change". - 2026-02 #589506: `committers()` loaded every diff commit per MR (60 s Puma kills); fixed by !223603 (SQL committer emails, flag `approval_committer_emails_from_diff`); side effect #590452 (primary CPU), then replica + per-diff cache. - 2026-03-16 !225446: traversal_ids JOIN behind `mr_lookup_by_checking_traversal_ids_on_join`; 1% -> 100% on gprd 2026-03-25/26, turned off the same day, per-group retries 03-27, removed in !238796 (2026-06-04). Rollback reason not recorded. - 2026-04/05 !231260: IN-operator (keyset CTE) optimisation behind `group_mr_in_operator_optimization`, only for group + include_subgroups + created_at sort + a single `state` + no other filter (the shape of 86% of >= 5 s calls). Trialled on gprd for single groups 2026-05-07/08 (six toggles), never a percentage rollout; removed in !244663 (2026-09-07, "FF has been disabled"), no written result. Review learnings: archived check into `array_scope`, `skip_default_order: true` in `paginate`, allowlist params, `banned_users` anti-join kept in the inner query. - 2026-06-05 !226653: 30 min response cache behind `cache_list_mr_on_group_api_responses`; enabled for 4 groups 2026-06-09, disabled 2026-06-12 after Redis incident gitlab-com/gl-infra/production#22302 (per-user key cardinality; "will not be rolled out further"); !242963 narrowed it to bot users; never re-enabled. - 2026-07 retro #606010: group list still number 1 error-budget contributor, two list endpoints = 47% of API apdex failures; conclusion: none of the options worked, little human-user impact, recalibrate targets (epic &23233). - 2026-08/09 !248316 (+ !253404): MATERIALIZED CTE for the unscoped label filter (GraphQL user-scope timeouts #603780); does not fire at group scope. ### In-flight work - 2026-09, Marc: - !257613 merged (loaded approval rules in the policy check: 63 queries / 145 ms per group page on gitlab-org). - !257173 open (skip X-Total behind `skip_total_count_on_group_merge_requests_api`; COUNT = 1,240 of 2,400 ms DB; open question whether dropping X-Total is a breaking change, rollout issue #630469 still named `cache_total_count...`). - !258522 / !258523 (user/project routes, drafts). - !257894, !258392 (committer/diff-commit preloads in GraphQL mergeability fields). - draft !230871 (skip rebase check for FF merge trains in GraphQL). - #628132 (assigned Marc: `mark_as_checking` UPDATE inside GETs). - 2026-09-30 epic &23858: Elasticsearch-backed `Search::AdvancedFinders::MergeRequestsFinder`, names `lib/api/merge_requests.rb` as a caller. ### Open issues - #591321 (3 root namespaces = 75% of slow calls, bots walking pages 1-100 every ~20 min, outreach stalled) - #602884 (view=simple recheck) - #592559 / #629383 (project list 500s via committers N+1) - #592339 (group GraphQL list + author filter 15 s) - #500577 (408 instead of 504) - #528570 (list vs single field inconsistency) - #423840 / #423924 (group count timeouts) - #580226 (user-scope 503 with 50k+ authored MRs) ### How other list endpoints avoid the costs - The group issues API pays the same COUNT and offset but has no mergeability work and preloads milestone project routes. - GraphQL `group.mergeRequests` keyset-paginates without a count, preloads per selected field (`head_pipeline`, `milestone: [:project, :group]`) and computes `detailedMergeStatus` only when asked (same service, so same per-node cost when asked). - The Vue group MR list fetches 20 rows by cursor, never asks for `detailedMergeStatus`, and caps state counts at 999. ### What to fix, ranked 1. Make `detailed_merge_status` cheap in list context (biggest per-request win; ~80% of serialize time, all Gitaly, ~90% of Redis on pages of open mergeable MRs). Without changing the response contract: - Pass list-context params from `MergeRequestBasic` into `DetailedMergeStatusService` so `CheckConflictStatusService` skips the live `source_branch_sha` Gitaly comparison (the `unchecked`/`checking` early returns already cover the post-push window that `MergeRequests::RefreshService` marks) - removes 1 Gitaly `FindCommit` per open mergeable MR (and the second one for ff/rebase projects via `skip_rebase_check`, precedent: EE merge trains and !230871). - Skip the ResultsStore Redis GET when a cacheable check is inactive for the project (LFS locks off, path locks unlicensed) and batch the remaining cached-result reads per page with MGET/pipelining (ResultsStore API change); memoise `temporarily_unapproved?` and `pipeline_creating?` per MR within the request. - Batch `branch_missing?` per project (one `branch_names` set per project instead of 4 Redis commands per MR), or drop it for MRs whose latest diff already knows the branches exist. - Preload `head_pipeline` (partition-aware) when the CI check is reachable. - Stop emitting the per-MR mergeability JSON log line and clock instrumentation in list context (100 lines per page; 13% + 7% of CPU samples). - Bigger lever, contract change (flag or `view=`): in list endpoints derive `detailed_merge_status` from the stored `merge_status` plus cached check results only, as the Vue list already does by never asking for it; docs already say lists "might not proactively update `merge_status`". Needs a decision on list/single divergence (#528570) and bot consumers (third-party polling integrations). - Expected: serialize phase 537 -> ~150-200 ms per 100 open MRs locally; on gitlab-org state=opened roughly the ~1.8 s non-DB share shrinks by more than half. 2. Fix the preload chain (small, no contract change): - Remove `.preload(:source_project, :target_project)` in `find_merge_requests` (it is redundant with `preload_routables` and breaks the nested route preload) and give `view=simple` `preload_routables` instead: -28 to -34 route queries per page. - Preload `milestone: [{ project: [:route, { namespace: :route }] }, { group: :route }]` as the Issue API does: -4 to -14 queries per page. - EE: preload `target_project: :ci_cd_settings` or make `skip_rebase_check?` lazy (only when the rebase check is reachable): -24 to -30 queries per page, paid today for merged MRs too. - EE: resolve the cascading merge-check settings (`only_allow_merge_if_*`, `has_linked_configurations?`) once per distinct project/ancestor instead of per MR, or preload `target_project: :group` + namespace settings: -15 to -20 queries per page on Premium/Ultimate. - EE: `missing_policy_approval_rules?` per MR -> use the already preloaded `approval_rules`/`approval_project_rules` (same pattern as !257613): -13 to -24 queries per page. - Trim `unresolved_notes` to the columns `mergeable_discussions_state?` reads (or an EXISTS/count) and skip it when neither project nor group requires resolved threads; drop preloads nothing on this route reads (`applicable_post_merge_approval_rules` tree, `approved_by_users`, `group: :saml_provider`, `namespace_settings_with_ancestors_inherited_settings`, `target_project: :project_feature`) after checking the other users of the shared scope (project/user routes, GraphQL). 3. Structural fix for `state=all` on large groups (the only thing that turns the 408s into 200s): an index-driven access path per project. Options: - (a) retry the IN-operator optimisation from !231260 as the engine of an opt-in keyset path (`pagination=keyset`, `without_count`, cursors so deep pages do not need OFFSET), allowlisted params, covering both a single `state` (index `(target_project_id, state_id, created_at, id)`) and `state=all` (`(target_project_id, created_at, id)`), `updated_at` after adding `(target_project_id, state_id, updated_at, id)`; first ask Saahmed/ahegyi why the May trial was abandoned. - (b) denormalise a namespace/traversal column or a read model (rejected by Database for the 342 GB table, multi-quarter). - (c) the Elasticsearch finder (&23858). - Recommend (a) as the next experiment, measured on postgres.ai against gitlab-org and on the GDK2 dataset. 4. Fix #602884 (`view=simple` enqueues `MergeabilityCheckBatchWorker` for every page because `serializer_options_for` leaves `skip_merge_status_recheck` nil; the project route's `recheck_mergeability_of` has the same hole) and pair it with #628132 (no `mark_as_checking` writes on GET). One-line fix plus spec. 5. Ship !257173 (skip X-Total) and follow with `pagination=keyset` for the MR list endpoints (`MergeRequest.supported_keyset_orderings` for created_at/updated_at/title, `paginate_with_strategies`, docs table); the count is 21-35% of wall time on gitlab-org state=opened pages and offset walkers (pages 1-100 every 20 min) are the main traffic. 6. Operational guard rails (process-heavy, lower priority): fail fast on shapes that always time out on huge groups (`state=all` or `search` without a narrowing filter above a cached visible-MR threshold) instead of burning 15 s of replica time per call; per-endpoint rate limit for bots; customer outreach (#591321). Do not revive the response cache (Redis incident, stale mergeability). ### Open questions and not measured - Why !231260 (IN-operator) and !225446 (traversal JOIN) were abandoned; no written result exists. - Production server-side percentiles and component split (db/redis/gitaly/cpu) for this run: Kibana needs SSO; the 83 X-Request-Ids and Discover links are in `prod/`. - Which statement is cancelled in each 408 (COUNT vs SELECT) - needs Kibana `json.db_duration_s` or Sentry QueryCanceled SQL. - Redis durations and memory locally (instrumentation off in dev); cold-cache cost of `ancestor?`/lock checks; the bot cache path; jira and security-policy-pipeline checks never active in the dataset. - Whether `approval_committer_emails_from_diff` is currently on for gitlab.com (yml default false after the 2026-07-30 reintroduction). ### Where everything lives and how to rerun - Durable copy (gitignored): `/Users/marcshaw/work/gdk2/gitlab/tmp/mrapi_investigation/` - `synthesis.md`, `common_brief.md` - `static/`: `trace.md`, `to_sql.out` - `dynamic/`: `report.md`, `dataset.md`, `seed.rb`, `trace.rb`, `curl_bench.sh`, `phases.rb`, `exposures.rb`, `callers.rb`, `profile.rb`, `plans.rb`, `plans/`, `trace/`, `trace_nobullet/`, `spec_probes/` - `prod/`: `report.md`, `timings.csv`, `request_ids.txt`, `kibana_urls.md`, `raw/` - `priorart/catalog.md` - `verify/route_preload.rb` - Original session scratchpad: `/private/tmp/claude-501/-Users-marcshaw-work-gdk2-gitlab/d2fe93b5-12a0-4e67-9469-da8984ffc66b/scratchpad/` (contains `dynamic/tokens.env`, not copied). - Throwaway specs: `/Users/marcshaw/work/gdk2/gitlab/tmp/mrapi_specs/` (run one at a time with `bundle exec rspec tmp/mrapi_specs/<file>_spec.rb`). - GDK2 dataset: group `mrapi-perf-bf9eb4` (id 4157), users `mrapi-dev` (3730), `mrapi-outsider` (3731). - Mint tokens in a runner: ``` u=User.find(3730); t=u.personal_access_tokens.create!(name: "mrapi-#{SecureRandom.hex(3)}", scopes: [:api], expires_at: 7.days.from_now, organization_id: u.organization_id); puts t.token ``` - Rerun, from `/Users/marcshaw/work/gdk2/gitlab` (the scripts set `GITALY_DISABLE_REQUEST_LIMITS=1` themselves; `MRAPI_RUNS` sets repetitions; results land next to the scripts in `trace/`, `phases*.json`, `plans/`): ``` MRAPI_NO_BULLET=1 bundle exec rails runner tmp/mrapi_investigation/dynamic/trace.rb '<scenario regex>' MRAPI_NO_BULLET=1 bundle exec rails runner tmp/mrapi_investigation/dynamic/phases.rb MRAPI_NO_BULLET=1 bundle exec rails runner tmp/mrapi_investigation/dynamic/exposures.rb bundle exec rails runner tmp/mrapi_investigation/dynamic/plans.rb bash tmp/mrapi_investigation/dynamic/curl_bench.sh [runs] [filter] ``` - The scripts read `dynamic/tokens.env` next to them; it was not copied, recreate it with the keys `MRAPI_DEV_TOKEN`, `MRAPI_OUTSIDER_TOKEN`, `MRAPI_ROOT_TOKEN`, `MRAPI_GROUP_ID=4157`, `MRAPI_GROUP_PATH=mrapi-perf-bf9eb4`, `MRAPI_DEV_ID=3730`, `MRAPI_OUTSIDER_ID=3731`. - Always compare with Bullet off (`MRAPI_NO_BULLET=1`); see `dynamic/report.md` section 2 for the other caveats. - GitLab.com probes: `prod/timings.csv` (83 rows), `prod/request_ids.txt`, Kibana Discover links in `prod/kibana_urls.md` (index pattern pubsub-rails-inf-gprd = 7092c4e2-4eb5-46f2-8305-a7da2edad090, filter `json.meta.caller_id:"GET /api/:version/groups/:id/merge_requests"`). - Kibana (needs Okta SSO; index pattern pubsub-rails-inf-gprd = `7092c4e2-4eb5-46f2-8305-a7da2edad090`): - All traffic, slowest first: https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-7d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.meta.caller_id:"GET /api/:version/groups/:id/merge_requests"'),columns:!(json.duration_s,json.db_duration_s,json.db_count,json.redis_calls,json.gitaly_calls,json.cpu_s,json.view_duration_s,json.status,json.params.value,json.username,json.correlation_id,json.params),sort:!(!(json.duration_s,desc))) - Slower than the 5 s target: https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-7d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.meta.caller_id:"GET /api/:version/groups/:id/merge_requests" and json.duration_s > 5'),columns:!(json.duration_s,json.db_duration_s,json.db_count,json.redis_calls,json.gitaly_calls,json.cpu_s,json.view_duration_s,json.status,json.params.value,json.username,json.correlation_id,json.params),sort:!(!(json.duration_s,desc))) - 408 / 5xx only: https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-7d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.meta.caller_id:"GET /api/:version/groups/:id/merge_requests" and json.status >= 400'),columns:!(json.duration_s,json.db_duration_s,json.db_count,json.redis_calls,json.gitaly_calls,json.cpu_s,json.view_duration_s,json.status,json.params.value,json.username,json.correlation_id,json.params),sort:!(!(json.duration_s,desc))) - Pre-built from earlier issues: slow list-MR requests dashboard https://log.gprd.gitlab.net/app/r/s/tkljo, Lens by root namespace https://log.gprd.gitlab.net/app/lens#/edit/10ba6226-c11a-4662-b8d6-99612dedd53d, Grafana API overview per-endpoint apdex https://dashboards.gitlab.net/d/api-main/api3a-overview?orgId=1&from=now-7d&to=now&var-environment=gprd&var-stage=main&viewPanel=panel-109 - KQL for the 2026-09-30 probes (each line below is `X-Request-Id scenario group run status total_s`; filter with `json.correlation_id : ("<id>" or ...)`): <details> <summary>X-Request-Ids of the 83 GitLab.com probe requests (2026-09-30 20:15-21:00 UTC)</summary> ``` a435ede28e7e9425-LHR default_pp100 g9970 r1 status=408 total=18.987947 a435ee65de9934ed-LHR default_pp100 g9970 r2 status=408 total=19.613574 a435eeecfd523edb-LHR per_page20 g9970 r1 status=408 total=17.516052 a435ef677f3ef65c-LHR per_page20 g9970 r2 status=408 total=19.607403 a435efef09e557a1-LHR page2 g9970 r1 status=408 total=18.400693 a435f06f2e0cd577-LHR page2 g9970 r2 status=408 total=15.416567 a435f0dd1d46d7c3-LHR page50 g9970 r1 status=408 total=17.610625 a435f1582accf64f-LHR page50 g9970 r2 status=408 total=18.252828 a435f1d729ad9533-LHR page100 g9970 r1 status=408 total=15.415659 a435f2448a0eef0d-LHR page100 g9970 r2 status=408 total=18.227605 a435f2c3a9854883-LHR state_opened g9970 r1 status=200 total=4.203031 a435f2eaf95d638e-LHR state_opened g9970 r2 status=200 total=4.177804 a435f3121e2c9461-LHR state_merged g9970 r1 status=408 total=15.588181 a435f3824940946d-LHR state_merged g9970 r2 status=408 total=15.738388 a435f3f1784f1471-LHR state_closed g9970 r1 status=408 total=17.668453 a435f46d2d2f001e-LHR state_closed g9970 r2 status=408 total=15.475967 a435f4db09e4e908-LHR scope_all g9970 r1 status=408 total=15.453786 a435f548cb5b9527-LHR scope_all g9970 r2 status=408 total=15.357122 a435f7baefa53d6d-LHR scope_created_by_me g9970 r1 status=200 total=3.214092 a435f7dc0b94940f-LHR scope_created_by_me g9970 r2 status=200 total=2.115163 a435f7f68e57bf02-LHR scope_assigned_to_me g9970 r1 status=200 total=3.214893 a435f8194fb9bd95-LHR scope_assigned_to_me g9970 r2 status=200 total=3.522368 a435f83cacf08560-LHR scope_reviews_for_me g9970 r1 status=200 total=7.852218 a435f87bf8d8f654-LHR scope_reviews_for_me g9970 r2 status=200 total=3.921434 a435f8a35d80be9f-LHR labels_backend g9970 r1 status=408 total=15.428624 a435f9116a9f1bf8-LHR labels_backend g9970 r2 status=408 total=15.415615 a435f97f39d4944e-LHR labels_backend_opened g9970 r1 status=200 total=3.148342 a435f9a01d81005f-LHR labels_backend_opened g9970 r2 status=200 total=3.737126 a435f9c57dd163e3-LHR not_labels_backend g9970 r1 status=408 total=19.654451 a435fa4d3a96beb3-LHR not_labels_backend g9970 r2 status=408 total=17.308799 a435fac67ef7955d-LHR labels_crw g9970 r1 status=200 total=0.948249 a435fad9cc3a4968-LHR labels_crw g9970 r2 status=200 total=1.243079 a435faee398ee8fa-LHR milestone_19_5 g9970 r1 status=200 total=4.425010 a435fb178ba7ef1d-LHR milestone_19_5 g9970 r2 status=200 total=4.505189 a435fb416c8afd82-LHR assignee_marc g9970 r1 status=200 total=2.006713 a435fb5bc895ed0a-LHR assignee_marc g9970 r2 status=200 total=2.416923 a435fb7979b878f2-LHR reviewer_marc g9970 r1 status=200 total=2.875669 a435fb991ecacd0a-LHR reviewer_marc g9970 r2 status=200 total=3.194262 a435fbb9fa721e12-LHR author_marc g9970 r1 status=200 total=3.273000 a435fbdbbed777c8-LHR author_marc g9970 r2 status=200 total=3.526662 a435fbff7a889408-LHR search_timeout g9970 r1 status=408 total=15.314557 a435fc6c3c6aa6f7-LHR search_timeout g9970 r2 status=408 total=15.367087 a435fcdaaebb94d5-LHR search_timeout_in_title g9970 r1 status=408 total=15.550768 a435fd492f74bec9-LHR search_timeout_in_title g9970 r2 status=408 total=15.421172 a435fdb6bc6a0032-LHR order_updated_at g9970 r1 status=408 total=15.509274 a435fe266f8237aa-LHR order_updated_at g9970 r2 status=408 total=15.393376 a435fe942b6eb29c-LHR order_title g9970 r1 status=408 total=15.378113 a435ff03284f6532-LHR order_title g9970 r2 status=408 total=15.583509 a435ff730872edeb-LHR order_milestone_due g9970 r1 status=408 total=19.332527 a435fffb3ee845b6-LHR order_milestone_due g9970 r2 status=408 total=15.376145 a4360073299a79c8-LHR order_priority g9970 r1 status=408 total=15.342027 a43600e21984b257-LHR order_priority g9970 r2 status=408 total=15.312492 a43601511a51c9ea-LHR order_label_priority g9970 r1 status=408 total=15.324053 a43601beb8debd76-LHR order_label_priority g9970 r2 status=408 total=15.320487 a436022c9a8abee4-LHR created_after_30d g9970 r1 status=200 total=5.001191 a436025a5cf6bdad-LHR created_after_30d g9970 r2 status=200 total=5.983668 a436028dcbcec4f1-LHR updated_after_7d g9970 r1 status=200 total=3.904136 a43602b3fa0b9401-LHR updated_after_7d g9970 r2 status=200 total=4.313948 a43602dd291bfc20-LHR merged_after_7d_merged g9970 r1 status=200 total=8.575171 a436031fed747193-LHR merged_after_7d_merged g9970 r2 status=200 total=9.893793 a436036b2ad7148c-LHR target_master_opened g9970 r1 status=200 total=3.171446 a436038c4ce8ef1f-LHR target_master_opened g9970 r2 status=200 total=2.292805 a43603a8aa9b4e1a-LHR draft_true_opened g9970 r1 status=200 total=2.727033 a43603c86d8294c1-LHR draft_true_opened g9970 r2 status=200 total=2.443536 a43603e5aa65cdc1-LHR with_labels_details g9970 r1 status=408 total=15.408948 a43604533a43cdba-LHR with_labels_details g9970 r2 status=408 total=15.424497 a43604c10d17330e-LHR view_simple g9970 r1 status=408 total=15.443216 a436052f4a5afc4d-LHR view_simple g9970 r2 status=408 total=15.627092 a436059e0848ef13-LHR non_archived_false g9970 r1 status=408 total=18.867561 a4360622be95d1fd-LHR non_archived_false g9970 r2 status=408 total=15.590325 a43606927c8c778c-LHR merge_status_recheck_ONCE g9970 r1 status=200 total=1.830813 a43606adb94dd184-LHR default_pp100 g6543 r1 status=408 total=20.135877 a43607393d5b93d7-LHR default_pp100 g6543 r2 status=408 total=15.478978 a43607a7ba8f955f-LHR default_pp100 g2750817 r1 status=200 total=8.163057 a43607e94bddbed5-LHR default_pp100 g2750817 r2 status=200 total=7.563620 a4360826dcca6430-LHR default_pp100 g12892338 r1 status=200 total=3.802947 a436084cbcb2d23c-LHR default_pp100 g12892338 r2 status=200 total=2.246284 a4360868b898d1fa-LHR graphql_opened_first100 g9970 r1 status=200 total=3.976282 a436088fbaecf77c-LHR graphql_opened_first100 g9970 r2 status=200 total=6.312615 a436093fade677a0-LHR search_timeout_opened_ONCE g9970 r1 status=200 total=6.772453 a43609785935cd96-LHR milestone_19_5_opened_ONCE g9970 r1 status=200 total=4.511172 a43609a1ca5963ef-LHR order_title_opened_ONCE g9970 r1 status=200 total=5.779006 a43609d40c49636d-LHR page49_opened_ONCE g9970 r1 status=200 total=2.996521 ``` </details> </details>