diff: Allow DiffStats to ignore whitespace changes

Add a whitespace_changes option to DiffStatsRequest, mirroring the one on CommitDiffRequest, so callers can get line counts under --ignore-all-space / --ignore-space-change without streaming a full CommitDiff. Consumed by gitlab!256363.

Rollout order across the epic (&23646): 1. !9274 (no flag, Gitaly release + gem and GITALY_SERVER_VERSION bump), then !256363 and its stacked !256402 (shared flag rapid_diffs_metadata_from_stored_diff_files). !256379 (flag diff_skip_redundant_charset_detection) and !256367 (flag diff_file_blob_id_metadata_only) are independent and can merge any time. With every flag off, behaviour is unchanged apart from the Feature.enabled? call.

Detailed context for AI agents

Why. Rapid Diffs in Rails has two endpoints (diffs_stats, diff_files_metadata) that never render a patch but need whitespace-aware line counts when the user hides whitespace changes (the default for anonymous users). DiffStats had no whitespace option, so Rails ran a full CommitDiff with whitespace_changes set just to read lines_added / lines_removed. That RPC is the slowest CommitDiff variant since !8539 (merged) added the second numstat pass, and it is where the production 500s on those endpoints originate.

Prior work (#7067 (closed), #7086 (closed), !8539 (merged)) moved counts into CommitDiff so the streaming endpoint could avoid DiffStats. This is the complement for callers that have no reason to fetch patches at all.

Change.

• proto/diff.proto: DiffStatsRequest.WhitespaceChanges enum (same values and semantics as CommitDiffRequest.WhitespaceChanges) and field whitespace_changes = 4. Unset keeps the current behaviour.
• internal/gitaly/service/diff/numstat.go: the shared numstat helper takes the option and appends --ignore-all-space or --ignore-space-change to git diff --numstat -z. DiffStats passes the request value.
• commit_diff.go: its call to numstat passes WHITESPACE_CHANGES_UNSPECIFIED explicitly. Its second pass exists to count whitespace-only files, so it must never ignore whitespace.
• proto/go/gitalypb/diff.pb.go regenerated with make proto. Unrelated *_grpc.pb.go comment-indentation churn from a newer protoc-gen-go-grpc was reverted.

Behaviour note. As with git diff -w, paths whose changes are entirely whitespace are omitted from the response. The Rails caller takes the file list from its stored diff files and treats a missing stats entry as 0/0, so no extra pass is needed here. Documented on the proto field.

Tests. TestDiffStats_whitespaceChanges covers unspecified, ignore and ignore-all against files with whitespace-only, amount-only and mixed changes. Existing TestDiffStats_* and TestCommitDiff_* unchanged.

Consolidated findings (2026-09-18, epic &23646)

Where the 500s came from. Both diffs_stats and diff_files_metadata fail inside Gitlab::GitalyClient::DiffStitcher while iterating a CommitDiff stream. An MR only takes that path when whitespace changes are hidden (DiffHelper#hide_whitespace? is true for every anonymous user and for users with show_whitespace_in_diffs off), when the diff files were cleaned from the DB (without_files), or for a commit_id. Gitlab::Git::Compare resolves both SHAs first, so the failing status is not a missing commit; the class is in the Sentry title (ResourceExhaustedError is already rescued globally).

What CommitDiff really streams. Gitaly runs git diff-tree -p over the whole range and its parser reads every patch byte; with whitespace hidden it runs a second git diff --numstat over the whole diff for line counts (!8539 (merged)). Patch bodies are sent in full up to the safe limits (100 files, 5000 lines, 500 KB), pruned to metadata past them, and cut off at the hard limits (1000 files, 50 000 lines, 5 MB). For MRs under 100 files, which is most, the two endpoints streamed and re-parsed the complete diff to read a handful of counts. Neither endpoint reads a patch byte: stats needs totals, count, real_size and overflow flags; metadata needs paths, flags, hashes, a blob id and per-file counts.

Why nobody had done this. #588880, #7067 (closed), #7086 (closed), !8539 (merged) and !223439 deliberately moved line counts into CommitDiff so the stream could drop DiffStats, on the assumption that the patch is fetched anyway. That holds for the stream and never held for these two endpoints. MergeRequestPresenter#include_diff_stats? still returned true for MRs, so DiffStats was never removed for them either.

Stored path costs found on the way. (1) Two CharlockHolmes charset scans per stored patch, in DiffCollection#expand_diff? and Gitlab::Git::Diff#encode_diff_to_utf8: 55% of wall time plus 28% GC on a 231-file MR, both skippable without changing results (!256379). (2) The diff column was selected for every row only to count lines and bytes for the overflow flags; SQL can return those counts (!256402). (3) code_review_id needs a blob id, stored rows carry none, so every request ran GetBlobs in batches of 250 with a 512 KB content limit, 5 RPCs and two thirds of the request on a 1000-file MR. Persisting to_id/from_id is blocked: merge_request_diff_files is in OverLimitTables and Migration/PreventAddingColumns rejects add_column. One FindChangedPaths RPC returns identical ids for every path in 25 to 45 ms (!256402). The metadata-only GetBlobs variant (!256367) was closed as redundant: Rapid Diffs is fully enabled and it only helped the legacy diffs page.

Renames. DiffStats already detects renames at git's default 50% similarity and reports old_path; CommitDiff uses --find-renames=30%. Only renames between the two thresholds differ, which is what the unless renamed_file? fallback in Gitlab::Diff::File#added_lines covers, so renamed rows keep their patch and #7428 (closed) was closed (50% is acceptable).

Rollout order and flags. !9274 first (no flag; needs a Gitaly release plus gitaly gem and GITALY_SERVER_VERSION bump). Then !256363, then !256402 stacked on it, both behind rapid_diffs_metadata_from_stored_diff_files (rollout #629875), Draft until the Gitaly release ships and the flag must stay off until that Gitaly is deployed, because an older Gitaly ignores the unknown proto field and returns whitespace-inclusive counts. !256379 is independent behind diff_skip_redundant_charset_detection (rollout #629894). With every flag off, behaviour is unchanged apart from the Feature.enabled? call.

Retiming with everything on (GDK, stack plus !256379, flags off vs on in the same process, medians of 8; stream = collection load without rendering):

files	whitespace	stats off / on	metadata off / on	stream load off / on	page total off / on
43	shown	50 / 13 ms	63 / 40 ms	74 / 54 ms	186 / 107 ms (43%)
43	hidden	63 / 12 ms	65 / 34 ms	116 / 82 ms	244 / 128 ms (47%)
231	shown	191 / 29 ms	211 / 58 ms	320 / 171 ms	722 / 257 ms (64%)
231	hidden	108 / 27 ms	110 / 54 ms	321 / 142 ms	540 / 223 ms (59%)
268	shown	139 / 35 ms	230 / 65 ms	603 / 519 ms	972 / 619 ms (36%)
268	hidden	130 / 32 ms	132 / 65 ms	278 / 162 ms	539 / 259 ms (52%)
1000	shown	192 / 95 ms	337 / 155 ms	493 / 396 ms	1022 / 646 ms (37%)
1000	hidden	236 / 95 ms	266 / 161 ms	666 / 539 ms	1168 / 795 ms (32%)

Gitaly RPCs per page load with whitespace hidden: 12 down to 7 (stats FindCommit x2 + CommitDiff to none; metadata FindCommit x2 + CommitDiff to one FindChangedPaths; the stream keeps its 6 because it needs the patches). With whitespace shown, stats and metadata make zero and one RPC instead of DiffStats plus up to five GetBlobs batches. The charset skip also speeds up the whitespace-hidden stream by 19 to 56%, since Gitaly patches go through the same second scan. First request per diff version with flags on pays one DiffStats (15 to 60 ms), cached for a week. Totals and per-file flags were verified identical to the old path on all four MRs.

Still open, in the epic. #629872 (Compare runs FindCommit twice for SHAs it already holds, on every stream and diff_file request), #629873 (fold diffs_stats into the metadata response), #629874 (fail soft on Gitaly errors, plus the product question of anonymous users defaulting to hidden whitespace, which routes all anonymous traffic to the live path). Outside Rapid Diffs, MergeRequest#diff_stats behind GraphQL diffStatsSummary runs DiffStats uncached on every MR widget load.

Closes #7427 Epic: &23646