Compute Code Review Analytics API line counts from diff stats

What does this MR do and why?

Related to https://gitlab.com/gitlab-org/gitlab/-/work_items/636159

Follow-up to !268230 (merged).

GET /api/v4/analytics/code_review preloads latest_merge_request_diff: :merge_request_diff_files, which loads every diff file row, with its full diff text, for up to 100 merge requests into one Puma worker. It uses them only to sum the added and removed lines. On GitLab.com, anonymous calls have already reached 516 MB at the default 20 per page.

Behind the code_review_analytics_api_diff_stats flag (gitlab_com_derisk, project actor):

  • The entity sums the diff stats instead of the diff files. It reads the same Gitlab::Diff::StatsCache (Redis) as the diff file collection, and on a miss calls MergeRequestDiff#diff_stats: one Gitaly DiffStats call, as today.
  • The endpoint calls with_code_review_api_entity_associations(preload_diff_files: false), so merge_request_diff_files isn't loaded. latest_merge_request_diff is still preloaded, because commits_count and diff_refs read its columns.

When the flag is off, nothing changes compared with !268230 (merged).

Do the counts match?

I compared the old and new diff_stats for an MR from every branch in the gitlab-test repository against master (about 90 MRs). All counts were the same, including renamed files (gitaly-rename-test), large and collapsed diffs (expand-collapse-*), binaries, LFS and symlinks.

The only difference was MRs with no merge base (orphaned-branch, snippet/*). With a blank base SHA, Repository#diff_stats returns empty stats, while the old path counted the diff lines. Those MRs keep using the diff files, so their counts stay the same too. The same fallback covers a failed or timed out DiffStats call, which also gives empty stats.

An MR with more files than the diff limits can still get higher counts, because the stats cover every file and the stored diff files are capped. The rollout should compare a sample of large MRs.

How to set up and validate locally

bundle exec rspec ee/spec/lib/api/entities/analytics/code_review/merge_request_spec.rb \
  ee/spec/requests/api/analytics/code_review_analytics_spec.rb
  • Entity spec: flag on gives the same 118 / 9 without calling diffs. With the stats cache warm there's no Gitaly call. An MR without a merge base, or with empty diff stats, counts the diff files. Flag off builds the collection once.
  • Request spec: with the flag on there's no merge_request_diff_files query, and no N+1 for 1 vs 3 MRs. With it off, the diff files are preloaded once. The "no diff files" example fails without the endpoint change.

In GDK, compare diff_stats in /api/v4/analytics/code_review?project_id=<id> with the flag off and then on:

Feature.enable(:code_review_analytics_api_diff_stats, Project.find(<id>))

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist.

Edited by Jatin Parihar

Merge request reports

Loading
Loading