Decide AI reviewable diff files from the patch alone

What does this MR do and why?

POST /api/:version/ai/duo_workflows/code_review/add_comments selects the files Duo Code Review can comment on with MergeRequest#ai_reviewable_diff_files. To tell text from binary, it batch-loads every old and new blob from Gitaly, up to 512 KB each. On large merge requests this allocates hundreds of megabytes to several gigabytes per request on the ai-assisted fleet, and it accounts for most of the request's Gitaly time.

Git has already made that call when it built the patch: binary files get a Binary files … differ notice instead of a patch. This MR adds Gitlab::Diff::File#ai_reviewable_patch?, which decides from the stored patch alone, so no blobs are loaded and the GetBlobs requests go away. The Duo Workflow path uses it behind the duo_code_review_skip_blob_loading feature flag (gitlab_com_derisk, project actor). ai_reviewable? and the legacy AI Gateway review path are unchanged, because that path sends blob contents to the LLM.

Git and the blob check disagree on a few kinds of file, so the new predicate adds two patch checks to keep today's results:

• A patch containing a NUL byte is skipped. This covers files like PDFs, which git diffs as text but the blob check treats as binary by their magic bytes.
• An LFS pointer patch is skipped. The Changes tab shows these files as stored in LFS, so there are no lines to comment on.

What still differs from today:

File	Today	This MR
Text file whose content starts with a magic number such as %PDF-1.5	Not reviewable	Reviewable (git diffs it as text)
LFS pointer with a text extension such as .md	Reviewable	Not reviewable

This MR started as an 8000-byte blob cap. Following review feedback, it now skips blob loading entirely, which is both simpler and faster (see the benchmark below).

Production evidence

Kibana, api_json, add_comments on the ai-assisted fleet, grouped by mem_total_bytes over 3 days:

Allocation	Requests	Total allocated	Avg Gitaly	Avg external HTTP calls	Avg CPU
< 32 MB	31,057	579 GB
32–128 MB	3,153	156 GB
128–512 MB	332	65 GB	0.65 s	46	4.7 s
> 512 MB	12	13 GB	3.39 s	77	12.1 s

The largest requests allocated 2,498 MB, 1,856 MB and 860 MB. The 860 MB one reviewed an MR with 371 files and 651 blobs at about 191 KB each on average after the 512 KB cap, which predicts about 850 MB.

Kibana links (pubsub-rails-inf-gprd*, sorted by json.mem_total_bytes):

• Largest add_comments requests, 26–29 Sep
• add_comments requests that allocated more than 512 MB, 26–29 Sep (this window ends later than the table above and includes the 9.8 GB requests, so its counts are slightly higher: 16 requests above 512 MB)
• The two 9.8 GB requests on 2026-09-29

On 2026-09-29 a single review of an MR with thousands of changed files allocated 9.8 GB twice (a 60 s timeout followed by a retry) and moved one pod from 6.6 GB to 8.6 GB. That request timed out inside FileExclusion#excluded_files, which this MR does not address; it is the first follow-up below.

Follow-ups

• Build excluded_files from diff paths instead of a second full diff collection.
• Stop at the first reviewable file where only presence is needed.

References

• #630829
• Rollout issue: #631315
• Kibana: largest add_comments requests, 26–29 Sep

Screenshots or screen recordings

Not a UI change. Local benchmark of ai_reviewable_diff_files, after a warm-up run. The 8000-byte cap column is the earlier version of this MR, for comparison.

MR	Files	Loading blobs (today)	8000-byte cap	Patch only (this MR)
400 files × 400 KB	400	2,241 MB, 10.0 s, 6 Gitaly calls	39 MB, 4.9 s, 6 calls	7 MB, 1.8 s, 2 calls
123 files × 100 KB	109	151 MB, 1.35 s, 3 calls	12 MB, 1.07 s, 3 calls	3 MB, 0.65 s, 2 calls
735 small files	724	44 MB, 2.0 s	45 MB, 2.0 s	15 MB, 1.2 s

The selected files are identical in all three. They only differ on an MR built from the edge cases in the table above.

How to set up and validate locally

1. Open a merge request with many large text files changed (for example, a few hundred files of a few hundred KB each), plus a PNG, a PDF and an LFS pointer.
2. In a Rails console, compare the two ways of selecting files:

   mr = MergeRequest.find(<id>)
   Gitlab::Memory::Instrumentation.with_memory_allocations { mr.ai_reviewable_diff_files.map(&:file_path) }
   
   mr = MergeRequest.find(<id>)
   Gitlab::Memory::Instrumentation.with_memory_allocations { mr.ai_reviewable_diff_files(patch_only: true).map(&:file_path) }

   mem_total_bytes drops, and the file lists match apart from the cases listed above.
3. To exercise the endpoint path, enable the flag:

   Feature.enable(:duo_code_review_skip_blob_loading, project)

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.