Fix quadratic line reading in CI trace reverse scan

Hordur Freyr Yngvasonrequested to merge
fix/quadratic-concat-in-trace-reverse-line into master

What does this MR do and why?

Gitlab::Ci::Trace::Stream#reverse_line and #reverse_line_with_max_size rebuild the accumulated partial-line buffer on every 4KB backward step ((buf + debris).each_line). Across trace segments that contain no newlines — carriage-return progress bars, base64 dumps — this makes reverse iteration quadratic in segment size, so scans of large traces burn seconds of CPU where milliseconds would do.

This path runs synchronously in Ci::BuildFinishedWorker (urgency :high) via Ci::Build#update_coverage → Stream#extract_coverage, which scans the whole trace backwards when the coverage regex never matches. It also serves job log reads via Stream#raw(last_lines:).

This MR unifies both methods into a single reverse_lines helper that accumulates chunks in an array and joins them once per newline flush, making the scan linear. Because a flush is triggered by a newline in the earliest chunk, the retained partial line stays under BUFFER_SIZE after each flush.

The output is unchanged: verified byte-identical against the previous implementation across randomized and edge-case traces (newline-free segments, newlines on exact buffer boundaries, trailing content without a newline, empty streams), for both methods and a range of max_size caps.

Not addressed here (possible follow-up): read_last_lines_with_max_size builds its result with result = line + result, which is also quadratic but bounded by max_size.

References

How to set up and validate locally

  1. Run the spec file:

    bin/rspec spec/lib/gitlab/ci/trace/stream_spec.rb

  2. Optionally, compare a reverse scan over a newline-free trace before and after this change in a Rails console — it completes near-instantly after, and noticeably slowly before:

    data = "\rprogress 42% " * 2_000_000 # ~26MB, no newlines
stream = Gitlab::Ci::Trace::Stream.new { StringIO.new(data) }
Benchmark.realtime { stream.raw(last_lines: 1) }

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports