Bound the size of rendered notebook diffs

What does this MR do and why?

The notebook (.ipynb) transform in gems/ipynbdiff can now refuse a notebook whose transformed text has more lines than the caller allows, and the notebook diff shows the raw .ipynb diff for such a file. The blob size limit bounds the notebook bytes, but a degenerate notebook with one character per line transforms to many more lines per byte than an ordinary notebook, and every step after the transform (symbol map, diff process, line mapping, highlighting, cache entry) grows with those lines. The limits apply behind the cap_rendered_notebook_size feature flag (gitlab_com_derisk, default off); disabling the flag restores the current behavior.

Design notes

Gem. IpynbDiff::Transformer.new(max_lines:, max_bytes:) (both nil by default, so the gem behaves as before) checks the transformed blocks after transform_document and before the symbol map parse, and raises IpynbDiff::TransformedNotebookTooLargeError, whose message and lines/bytes readers carry the measured counts. max_bytes counts the bytes of the transformed lines; the app does not pass it (see the limits below) and it stays as a gem option. IpynbDiff.diff(..., limits: {}) forwards the options to the transformer (Metrics/ParameterLists is at its maximum, so they go in one keyword). IpynbDiff.diff transforms from fully (transform and symbol map parse) before it transforms to, so when only to is over the limit, one transform of at most 512 KB (about 0.3 s) is spent before the error. The gem version stays 0.4.8: the in-repo gems do not bump their version on change (the last five ipynbdiff changes did not), and the app uses the path gem without a Gemfile.lock change.

GitLab. Gitlab::Diff::Rendered::Notebook::DiffFile::RENDERED_LIMITS = { max_lines: 100_000 } is passed when the flag is on. The error is rescued like InvalidNotebookError: the file logs IPYNB_DIFF_OVER_LIMITS with the measured lines and bytes, so the limit can be tuned from the logs, and shows the raw diff. DiffCache#render_and_write maps the error to the existing too_large marker (1 hour), so the notebook is not transformed again while the marker lives and later requests log IPYNB_DIFF_CACHE_SKIPPED with skip_reason: too_large, as for a rendered diff over MAX_BYTES. The transform output of notebooks under the limits does not change, so DiffCache::VERSION stays 1.

Why this limit (measured locally on synthetic notebooks of about 512 KB, the most Gitlab::Diff::File#fetch_blob reads):

Notebook of ~512 KB Transformed lines Transformed KB Transform time
Code cells with 40-character lines about 15,000 592 0.03 s
Empty raw cells 49,146 463 0.10 s
Empty code cells 56,980 623 0.18 s
One code cell with one-character lines 87,000 to 131,000 256 0.29 s
One markdown cell with one-character lines 175,000 to 262,000 512 0.16 s

The limit of 100,000 lines leaves about 6x headroom over a 512 KB notebook of ordinary code and catches the degenerate one-character-per-line shapes; notebooks of empty cells stay under it. Transform timings for notebooks near the limit: 0.35 s for 82k lines, 0.44 s for 99k, 0.58 s for 131k; the capped path (transform, size check, error) takes 0.11 s. The transform does not multiply bytes (the output is at most about the input size, 426 KB at most across 13 shapes measured at 511 KB), so no byte limit is passed: with a 512 KB blob limit it would be out of reach, and DiffCache::MAX_BYTES already bounds the cache entry.

Related hardening. https://gitlab.com/gitlab-org/gitlab/-/issues/607080 tracks the parser-side guards (byte cap and nesting depth before the JSON parse). This MR bounds the transform output; the parser details stay there.

Specs. gems/ipynbdiff/spec/ipynb_diff/transformer_spec.rb (limits at, below and without the fixture's size; no symbol map parse over the limit) and gems/ipynbdiff/spec/ipynb_diff_spec.rb (limits: through IpynbDiff.diff). spec/lib/gitlab/diff/rendered/notebook/diff_file_spec.rb: over the limits the file shows the raw diff and logs the fallback with the counts, the flag-off path renders, and with the cache the second file of the same blobs does not transform again. spec/lib/gitlab/diff/rendered/notebook/diff_cache_spec.rb: the error leaves the too_large marker with the 1 hour TTL.

Feature flag, rollout, parent MRs, sibling MR, ownership
  • Feature flag: cap_rendered_notebook_size (gitlab_com_derisk, project actor, default off). Rollout issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/638641. Rollback: disable the flag; the transformer then receives no limits. A too_large marker written while the flag was on keeps the raw diff for that blob pair for up to 1 hour after the flag is disabled (marker reads do not check the flag). No changelog entry: the flag is off by default.
  • Parents: !263925 (merged) (render cache), !263894 (merged) (reduce_notebook_diff_rendering), !263933 (merged) (stop_notebook_text_diff_at_timeout).
  • Sibling from the same follow-up issue: !268419 (merged) batches the render cache reads per file collection. It changes DiffCache#fetch and the file collections; this MR changes DiffCache#render_and_write and DiffFile, so the merge order does not matter.
  • Ownership: gems/ipynbdiff and Gitlab::Diff::Rendered::Notebook belong to MLOps. The diff endpoints that call them belong to Code Review (code_review_workflow), so Code Review should co-review.
  • No database changes.

References

Screenshots or screen recordings

No UI change for notebooks under the limits. A notebook over the limits shows the raw .ipynb diff, as a notebook whose render times out does today.

How to set up and validate locally

  1. In the Rails console, build a notebook with 101,000 empty source lines (about 394 KB, under the 512 KB blob limit; 101,005 transformed lines) and a copy that changes the last line, so the raw diff stays small enough to render, and commit both to a GDK project:

    nb = ->(last) { JSON.generate({ cells: [{ cell_type: 'code', metadata: {}, outputs: [], source: Array.new(100_999, '') + [last] }], metadata: {}, nbformat: 4, nbformat_minor: 5 }, array_nl: "\n") }
    project = Project.find_by_full_path('<namespace>/<project>')
    user = project.first_owner
    project.repository.commit_files(user, branch_name: 'lines', start_branch_name: project.default_branch, message: 'Add', actions: [{ action: :create, file_path: 'lines.ipynb', content: nb.('a') }])
    project.repository.commit_files(user, branch_name: 'lines', message: 'Change', actions: [{ action: :update, file_path: 'lines.ipynb', content: nb.('b') }])
  2. In a terminal in the GDK gitlab directory, run tail -f log/development_json.log | grep IPYNB_DIFF_.

  3. Open http://gdk.test:3000/<namespace>/<project>/-/commit/<sha of the "Change" commit>. Expect: the rendered notebook (one changed line a to b at the end of the code cell), and an IPYNB_DIFF_GENERATED line.

  4. In the Rails console, run Feature.enable(:cap_rendered_notebook_size) and Rails.cache.clear, then reload the commit page. Expect: the raw .ipynb diff (one changed JSON line), and an IPYNB_DIFF_OVER_LIMITS line with lines: 101005.

  5. Reload the page. Expect: the raw diff, and an IPYNB_DIFF_CACHE_SKIPPED line with skip_reason: too_large instead of a new transform.

  6. Repeat steps 1 to 4 with Array.new(999, ''). Expect: the rendered notebook with the flag on and off.

Checked locally through Gitlab::Diff::FileCollection::Commit with these notebooks: flag off renders (10,004 rendered diff lines), flag on returns the raw diff and logs IPYNB_DIFF_OVER_LIMITS with lines: 101005, the second request logs the skipped marker, and the 1,000-line notebook renders with the flag on.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Eduardo Bonet

Merge request reports

Loading
Loading