Skip notebook rendering where diff responses do not use it

What does this MR do and why?

Merge request and compare diff responses that contain no diff lines no longer render notebook (.ipynb) diffs, and the stored diff batch no longer renders each notebook only to read its file paths. Each notebook render runs the IpynbDiff transform and a diff subprocess, and nothing caches the result, so these endpoints repeated that work on every request. All changes are behind the reduce_notebook_diff_rendering feature flag (gitlab_com_derisk, default off); disabling the flag restores the current behavior.

Design notes

Three changes, one flag (project actor):

  1. DiffHelper#diff_options sets use_extra_viewer_as_main: false for the diffs_metadata, diff_files_metadata and diffs_stats actions. Gitlab::Diff::FileCollection::Base#decorate_diff! calls has_renderable? (which runs the notebook diff) only when that option is true. This one place covers Projects::MergeRequests::DiffsController#diffs_metadata and the rapid diffs RapidDiffs::Resource#diff_files_metadata and #diffs_stats of the merge request, compare and new merge request controllers. The commit controller already passes false (commit_diff_options).
  2. Gitlab::Diff::FileCollection::Base#diff_paths reads the old and new paths from the raw diffs, instead of diff_files.flat_map(&:paths), which decorated (and rendered) every file. It returns the same set in the same order. diff_file_paths is not reused because it drops the old path of a renamed file, and the repository comparison filters by these paths. Callers: MergeRequestDiff#diffs_in_batch (on the repository comparison path it runs before the stale? check, so a conditional GET also rendered every notebook of the page), #diffs_for_streaming and #paginated_diffs.
  3. MergeRequestDiff#paginated_diffs passes the raw stored diffs to generated_files on the repository comparison path, instead of the decorated (and rendered) files. Gitlab::Git::ChangedPath.from_diff reads only the file status, the paths and the modes, which Gitlab::Git::Diff has. This path builds the first diff files of the rapid diffs merge request page when whitespace changes are ignored.

Visible difference with the flag on. DiffFileMetadataEntity and RapidDiffs::DiffsStatsEntity expose added_lines and removed_lines. A rendered notebook file has no diff stats, so it counts the lines of the rendered diff. Without rendering, these endpoints report the stats of the raw .ipynb diff. For merge requests with notebooks, the file tree and the total line counts can then differ from the counts in the file header, which still comes from the rendered diff in diffs_batch and diffs_stream. Paths, file hashes, code_review_id, file flags and conflict types do not change; the request spec compares them for diffs_metadata with the flag on and off, and pins the raw and rendered line counts for a notebook whose cell output changes. The commit page already shows raw counts for notebooks. Keeping the rendered counts needs the render cache below.

Follow-ups in sibling MRs (rollout issue https://gitlab.com/gitlab-org/gitlab/-/work_items/635151):

  • !263925 (merged) caches the rendered notebook diff per blob pair (cache_notebook_diffs), so the stored-diff path of diffs_batch, diffs_stream and DiffNote#fetch_diff_file (app/models/diff_note.rb:176) stop rendering on every request.
  • !263933 (merged) stops the text diff subprocess at the render timeout (stop_notebook_text_diff_at_timeout). IpynbDiff.diff returns a lazy Diffy::Diff, so its diff subprocess runs outside Gitlab::RenderTimeout, and Open3.capture3 waits for the child in its ensure block.

Ownership: the controllers, DiffHelper and the file collections belong to code_review_workflow; Gitlab::Diff::Rendered::Notebook belongs to MLOps. Code Review should co-review.

No sibling MR touches these files. No database changes: no queries change.

References

Screenshots or screen recordings

No layout change. With the flag on, the file tree and the total line counts of notebook files can show different numbers (see the design notes).

How to set up and validate locally

  1. In any GDK project, upload a Jupyter notebook to the default branch (+ > Upload file), then edit one cell's source on a new branch and click "Create merge request".
  2. In a terminal in the GDK gitlab directory, run tail -f log/application_json.log | grep IPYNB_DIFF_GENERATED.
  3. Open http://gdk.test:3000/<namespace>/<project>/-/merge_requests/<iid>/diffs_metadata.json. Expect: one IPYNB_DIFF_GENERATED line per changed notebook.
  4. In the Rails console, run Feature.enable(:reduce_notebook_diff_rendering).
  5. Reload the diffs_metadata.json URL. Expect: no new IPYNB_DIFF_GENERATED line, and the same files in diff_files.
  6. Open the merge request's Changes tab. Expect: the notebook still shows the rendered view.
More endpoints to check

With the flag on, these also log no IPYNB_DIFF_GENERATED line:

  • /-/merge_requests/<iid>/diff_files_metadata and /-/merge_requests/<iid>/diffs_stats (rapid diffs)
  • /-/compare/diff_files_metadata?from=<base>&to=<branch> and /-/compare/diffs_stats?from=<base>&to=<branch>

While signed out, /-/merge_requests/<iid>/diffs?rapid_diffs=true logs one line per notebook instead of two.

/-/merge_requests/<iid>/diffs_batch.json?page=0&per_page=30 while signed out (whitespace changes ignored, so the repository comparison path) logs one line per notebook instead of two, and a repeated request that returns 304 Not Modified logs none instead of one per notebook. Signed in, it logs one line per notebook with the flag on or off.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Eduardo Bonet

Merge request reports

Loading
Loading