Skip notebook rendering where diff responses do not use it
What does this MR do and why?
Merge request and compare diff responses that contain no diff lines no longer render notebook (.ipynb) diffs, and the stored diff batch no longer renders each notebook only to read its file paths. Each notebook render runs the IpynbDiff transform and a diff subprocess, and nothing caches the result, so these endpoints repeated that work on every request. All changes are behind the reduce_notebook_diff_rendering feature flag (gitlab_com_derisk, default off); disabling the flag restores the current behavior.
Design notes
Three changes, one flag (project actor):
DiffHelper#diff_optionssetsuse_extra_viewer_as_main: falsefor thediffs_metadata,diff_files_metadataanddiffs_statsactions.Gitlab::Diff::FileCollection::Base#decorate_diff!callshas_renderable?(which runs the notebook diff) only when that option is true. This one place coversProjects::MergeRequests::DiffsController#diffs_metadataand the rapid diffsRapidDiffs::Resource#diff_files_metadataand#diffs_statsof the merge request, compare and new merge request controllers. The commit controller already passesfalse(commit_diff_options).Gitlab::Diff::FileCollection::Base#diff_pathsreads the old and new paths from the raw diffs, instead ofdiff_files.flat_map(&:paths), which decorated (and rendered) every file. It returns the same set in the same order.diff_file_pathsis not reused because it drops the old path of a renamed file, and the repository comparison filters by these paths. Callers:MergeRequestDiff#diffs_in_batch(on the repository comparison path it runs before thestale?check, so a conditional GET also rendered every notebook of the page),#diffs_for_streamingand#paginated_diffs.MergeRequestDiff#paginated_diffspasses the raw stored diffs togenerated_fileson the repository comparison path, instead of the decorated (and rendered) files.Gitlab::Git::ChangedPath.from_diffreads only the file status, the paths and the modes, whichGitlab::Git::Diffhas. This path builds the first diff files of the rapid diffs merge request page when whitespace changes are ignored.
Visible difference with the flag on. DiffFileMetadataEntity and RapidDiffs::DiffsStatsEntity expose added_lines and removed_lines. A rendered notebook file has no diff stats, so it counts the lines of the rendered diff. Without rendering, these endpoints report the stats of the raw .ipynb diff. For merge requests with notebooks, the file tree and the total line counts can then differ from the counts in the file header, which still comes from the rendered diff in diffs_batch and diffs_stream. Paths, file hashes, code_review_id, file flags and conflict types do not change; the request spec compares them for diffs_metadata with the flag on and off, and pins the raw and rendered line counts for a notebook whose cell output changes. The commit page already shows raw counts for notebooks. Keeping the rendered counts needs the render cache below.
Follow-ups in sibling MRs (rollout issue https://gitlab.com/gitlab-org/gitlab/-/work_items/635151):
- !263925 (merged) caches the rendered notebook diff per blob pair (
cache_notebook_diffs), so the stored-diff path ofdiffs_batch,diffs_streamandDiffNote#fetch_diff_file(app/models/diff_note.rb:176) stop rendering on every request. - !263933 (merged) stops the text diff subprocess at the render timeout (
stop_notebook_text_diff_at_timeout).IpynbDiff.diffreturns a lazyDiffy::Diff, so itsdiffsubprocess runs outsideGitlab::RenderTimeout, andOpen3.capture3waits for the child in itsensureblock.
Ownership: the controllers, DiffHelper and the file collections belong to code_review_workflow; Gitlab::Diff::Rendered::Notebook belongs to MLOps. Code Review should co-review.
No sibling MR touches these files. No database changes: no queries change.
References
- Related to https://gitlab.com/gitlab-org/gitlab/-/work_items/635044
- Feature flag rollout issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/635109
- Follow-ups: !263925 (merged), !263933 (merged)
Screenshots or screen recordings
No layout change. With the flag on, the file tree and the total line counts of notebook files can show different numbers (see the design notes).
How to set up and validate locally
- In any GDK project, upload a Jupyter notebook to the default branch (
+ > Upload file), then edit one cell'ssourceon a new branch and click "Create merge request". - In a terminal in the GDK
gitlabdirectory, runtail -f log/application_json.log | grep IPYNB_DIFF_GENERATED. - Open
http://gdk.test:3000/<namespace>/<project>/-/merge_requests/<iid>/diffs_metadata.json. Expect: oneIPYNB_DIFF_GENERATEDline per changed notebook. - In the Rails console, run
Feature.enable(:reduce_notebook_diff_rendering). - Reload the
diffs_metadata.jsonURL. Expect: no newIPYNB_DIFF_GENERATEDline, and the same files indiff_files. - Open the merge request's Changes tab. Expect: the notebook still shows the rendered view.
More endpoints to check
With the flag on, these also log no IPYNB_DIFF_GENERATED line:
/-/merge_requests/<iid>/diff_files_metadataand/-/merge_requests/<iid>/diffs_stats(rapid diffs)/-/compare/diff_files_metadata?from=<base>&to=<branch>and/-/compare/diffs_stats?from=<base>&to=<branch>
While signed out, /-/merge_requests/<iid>/diffs?rapid_diffs=true logs one line per notebook instead of two.
/-/merge_requests/<iid>/diffs_batch.json?page=0&per_page=30 while signed out (whitespace changes ignored, so the repository comparison path) logs one line per notebook instead of two, and a repeated request that returns 304 Not Modified logs none instead of one per notebook. Signed in, it logs one line per notebook with the flag on or off.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.