diffs_stats and diff_files_metadata on the Rapid Diffs MR page were 500ing in production because both endpoints route through Gitaly's CommitDiff patch stream, even though neither one reads a single patch byte. This epic moves both to serve from stored merge_request_diff_files plus at most one cheap Gitaly RPC, removes the CommitDiff dependency that caused the 500s, and picks up the pre-existing stored-path inefficiencies found while doing that work. <details> <summary>Detailed context for AI agents</summary> Both endpoints only need cheap data: diffs_stats needs added/removed totals, file count, real_size, overflow flags. diff_files_metadata needs per-file paths, flags, hashes, a blob id for code_review_id, and per-file line counts. Neither needs patch text. The CommitDiff path is only taken when whitespace changes are hidden (default for anonymous users and for users with the preference off), when diff files were cleaned from the DB (without_files), or for a single commit_id. Background: issue gitlab-org/gitlab#588880, gitaly#7067, gitaly#7086, gitaly!8539 and gitlab!223439 deliberately moved line counts into CommitDiff so the stream could drop DiffStats. Nobody looked at these two patch-less endpoints at the time, and a whitespace option on DiffStats was never considered. Gitaly's DiffStats is `git diff --numstat -z` with no whitespace option and git's default 50% rename detection; CommitDiff uses `--find-renames=30%` and honours whitespace_changes. Rollout order and flags: - gitaly!9274 first, no flag. Needs a Gitaly release plus a gitaly gem and GITALY_SERVER_VERSION bump in gitlab-org/gitlab. - Then !256363, followed by !256402 stacked on it. Both gated by the shared flag `rapid_diffs_metadata_from_stored_diff_files` (rollout issue gitlab-org/gitlab#629875). Draft until the Gitaly release ships. - !256379 (flag `diff_skip_redundant_charset_detection`, rollout gitlab-org/gitlab#629894) is independent and ready any time. !256367 was closed as redundant (legacy diffs page only). - With every flag off, behaviour is unchanged apart from the Feature.enabled? call. Child work items and dependency chain: - gitlab-org/gitaly#7427 (gitaly!9274, gitlab-org/gitaly): add whitespace_changes to DiffStatsRequest. Must release before gitlab-org/gitlab#629868 can merge. - gitlab-org/gitlab#629868 (gitlab!256363): reuse stored diff files for diffs_metadata when whitespace is hidden, behind flag rapid_diffs_stored_diff_files_ignore_whitespace. Blocked on gitlab-org/gitaly#7427's Gitaly release plus a gitaly gem bump and GITALY_SERVER_VERSION bump. - gitlab-org/gitlab#629869 (gitlab!256402): load diff_files_metadata without fetching patch text. Stacked on gitlab-org/gitlab#629868. - gitlab-org/gitlab#629870 (gitlab!256379): skip charset detection where it can't change the outcome. Independent, no flag. - gitlab-org/gitlab#629871 (gitlab!256367): resolve blob_id without loading blob content. Closed as redundant: Rapid Diffs is fully enabled and !256402 covers its metadata endpoint, so it only helped the legacy diffs page. - gitlab-org/gitaly#7428: rename detection in Gitaly DiffStats. Closed, not needed: DiffStats already detects renames at git's default 50%. - gitlab-org/gitlab#629872: skip redundant FindCommit calls in Gitlab::Git::Compare when OIDs are already resolved. Follow-up. - gitlab-org/gitlab#629873: merge diffs_stats into diff_files_metadata (frontend + backend). Follow-up. - gitlab-org/gitlab#629874: fail soft instead of 500 on Gitaly errors in both endpoints, plus the anonymous-whitespace-default product question. Follow-up. Measured on GDK (medians per request, whitespace hidden) for MRs of 43 / 231 / 268 / 1000 files: - Original live path: diffs_stats 80 / 116 / 132 / 250 ms, diff_files_metadata 70 / 126 / 131 / 258 ms, RPCs: FindCommit x2 + CommitDiff (+ GetBlobs for metadata). - After gitlab-org/gitaly#7427 through gitlab-org/gitlab#629871: diffs_stats 11 / 25 / 28 / 89 ms with zero Gitaly calls; diff_files_metadata 40 / 57 / 63 / 144 ms with exactly one FindChangedPaths RPC. **Consolidated findings (2026-09-18, epic gitlab-org&23646)** *Where the 500s came from.* Both `diffs_stats` and `diff_files_metadata` fail inside `Gitlab::GitalyClient::DiffStitcher` while iterating a `CommitDiff` stream. An MR only takes that path when whitespace changes are hidden (`DiffHelper#hide_whitespace?` is true for every anonymous user and for users with `show_whitespace_in_diffs` off), when the diff files were cleaned from the DB (`without_files`), or for a `commit_id`. `Gitlab::Git::Compare` resolves both SHAs first, so the failing status is not a missing commit; the class is in the Sentry title (`ResourceExhaustedError` is already rescued globally). *What CommitDiff really streams.* Gitaly runs `git diff-tree -p` over the whole range and its parser reads every patch byte; with whitespace hidden it runs a second `git diff --numstat` over the whole diff for line counts (gitaly!8539). Patch bodies are sent in full up to the safe limits (100 files, 5000 lines, 500 KB), pruned to metadata past them, and cut off at the hard limits (1000 files, 50 000 lines, 5 MB). For MRs under 100 files, which is most, the two endpoints streamed and re-parsed the complete diff to read a handful of counts. Neither endpoint reads a patch byte: stats needs totals, count, `real_size` and overflow flags; metadata needs paths, flags, hashes, a blob id and per-file counts. *Why nobody had done this.* #588880, gitaly#7067, gitaly#7086, gitaly!8539 and !223439 deliberately moved line counts into `CommitDiff` so the stream could drop `DiffStats`, on the assumption that the patch is fetched anyway. That holds for the stream and never held for these two endpoints. `MergeRequestPresenter#include_diff_stats?` still returned true for MRs, so `DiffStats` was never removed for them either. *Stored path costs found on the way.* (1) Two CharlockHolmes charset scans per stored patch, in `DiffCollection#expand_diff?` and `Gitlab::Git::Diff#encode_diff_to_utf8`: 55% of wall time plus 28% GC on a 231-file MR, both skippable without changing results (!256379). (2) The `diff` column was selected for every row only to count lines and bytes for the overflow flags; SQL can return those counts (!256402). (3) `code_review_id` needs a blob id, stored rows carry none, so every request ran `GetBlobs` in batches of 250 with a 512 KB content limit, 5 RPCs and two thirds of the request on a 1000-file MR. Persisting `to_id`/`from_id` is blocked: `merge_request_diff_files` is in `OverLimitTables` and `Migration/PreventAddingColumns` rejects `add_column`. One `FindChangedPaths` RPC returns identical ids for every path in 25 to 45 ms (!256402). The metadata-only `GetBlobs` variant (!256367) was closed as redundant: Rapid Diffs is fully enabled and it only helped the legacy diffs page. *Renames.* `DiffStats` already detects renames at git's default 50% similarity and reports `old_path`; `CommitDiff` uses `--find-renames=30%`. Only renames between the two thresholds differ, which is what the `unless renamed_file?` fallback in `Gitlab::Diff::File#added_lines` covers, so renamed rows keep their patch and gitaly#7428 was closed (50% is acceptable). *Rollout order and flags.* gitaly!9274 first (no flag; needs a Gitaly release plus gitaly gem and `GITALY_SERVER_VERSION` bump). Then !256363, then !256402 stacked on it, both behind `rapid_diffs_metadata_from_stored_diff_files` (rollout #629875), Draft until the Gitaly release ships and the flag must stay off until that Gitaly is deployed, because an older Gitaly ignores the unknown proto field and returns whitespace-inclusive counts. !256379 is independent behind `diff_skip_redundant_charset_detection` (rollout #629894). With every flag off, behaviour is unchanged apart from the `Feature.enabled?` call. *Retiming with everything on* (GDK, stack plus !256379, flags off vs on in the same process, medians of 8; stream = collection load without rendering): | files | whitespace | stats off / on | metadata off / on | stream load off / on | page total off / on | |---|---|---|---|---|---| | 43 | shown | 50 / 13 ms | 63 / 40 ms | 74 / 54 ms | 186 / 107 ms (43%) | | 43 | hidden | 63 / 12 ms | 65 / 34 ms | 116 / 82 ms | 244 / 128 ms (47%) | | 231 | shown | 191 / 29 ms | 211 / 58 ms | 320 / 171 ms | 722 / 257 ms (64%) | | 231 | hidden | 108 / 27 ms | 110 / 54 ms | 321 / 142 ms | 540 / 223 ms (59%) | | 268 | shown | 139 / 35 ms | 230 / 65 ms | 603 / 519 ms | 972 / 619 ms (36%) | | 268 | hidden | 130 / 32 ms | 132 / 65 ms | 278 / 162 ms | 539 / 259 ms (52%) | | 1000 | shown | 192 / 95 ms | 337 / 155 ms | 493 / 396 ms | 1022 / 646 ms (37%) | | 1000 | hidden | 236 / 95 ms | 266 / 161 ms | 666 / 539 ms | 1168 / 795 ms (32%) | Gitaly RPCs per page load with whitespace hidden: 12 down to 7 (stats `FindCommit` x2 + `CommitDiff` to none; metadata `FindCommit` x2 + `CommitDiff` to one `FindChangedPaths`; the stream keeps its 6 because it needs the patches). With whitespace shown, stats and metadata make zero and one RPC instead of `DiffStats` plus up to five `GetBlobs` batches. The charset skip also speeds up the whitespace-hidden stream by 19 to 56%, since Gitaly patches go through the same second scan. First request per diff version with flags on pays one `DiffStats` (15 to 60 ms), cached for a week. Totals and per-file flags were verified identical to the old path on all four MRs. *Still open, in the epic.* #629872 (`Compare` runs `FindCommit` twice for SHAs it already holds, on every stream and `diff_file` request), #629873 (fold `diffs_stats` into the metadata response), #629874 (fail soft on Gitaly errors, plus the product question of anonymous users defaulting to hidden whitespace, which routes all anonymous traffic to the live path). Outside Rapid Diffs, `MergeRequest#diff_stats` behind GraphQL `diffStatsSummary` runs `DiffStats` uncached on every MR widget load. </details>