diff: Return line counts even when whitespace is ignored

Closes #7086 (closed)

We recently [1] introduced the ability to return the count of added and removed lines as part of the CommitDiff response. The goal was to save callers from having to invoke the DiffStats RPC solely to retrieve line counts.

There was one case in which line counts were unable to be returned -- when callers ask for whitespace to be ignored. This is because we would execute git-diff(1) with whitespace ignoring flags in effect, so it would never produce the necessary patch bodies for us to compute the counts. This ultimately meant that a call to DiffStats was necessary, if the user decided to ignore whitespace content using the toggle button in the MR diffs page.

We can eat extra latency overhead in CommitDiff if we want clients to avoid having to make the extra DiffStats call. We execute git-diff --numstat on their behalf, store the results, and then combine them with the contents of the diffManifest. This has the net effect of sending proper line counts even if the entire file was comprised of whitespace changes.

This way, CommitDiff can return correct line counts for the following diffs:

1. When the diff contains only whitespace changes.
2. When the diff contains no whitespace changes.
3. When the diff contains a mix of whitespace and functional changes.

Running the CommitDiff benchmark does show a marked increase in latency however:

$ go test -bench=CommitDiff -benchtime=40s ./internal/gitaly/service/diff
goos: linux
goarch: amd64
pkg: gitlab.com/gitlab-org/gitaly/v18/internal/gitaly/service/diff
cpu: Intel(R) Core(TM) Ultra 7 165H
BenchmarkCommitDiff/all_changes-22                    24        2084723223 ns/op
BenchmarkCommitDiff/whitespace_ignored-22             10        4472688268 ns/op
PASS
ok      gitlab.com/gitlab-org/gitaly/v18/internal/gitaly/service/diff   187.024s

Previously:

$ go test -bench=CommitDiff -benchtime=40s ./internal/gitaly/service/diff
goos: linux
goarch: amd64
pkg: gitlab.com/gitlab-org/gitaly/v18/internal/gitaly/service/diff
cpu: Intel(R) Core(TM) Ultra 7 165H
BenchmarkCommitDiff/all_changes-22                    25        1865013680 ns/op
BenchmarkCommitDiff/whitespace_ignored-22             12        3875419050 ns/op
PASS
ok      gitlab.com/gitlab-org/gitaly/v18/internal/gitaly/service/diff   107.576s

Which is a jump from ~3.8 seconds to ~4.4 seconds on a large diff, when whitespace ignoring is specified. This might be okay, as it does save the client from having to make yet another network round trip.

[1] !8462 (merged)