Skip to content

Skip timestamps in log if present when generating HTML

Pedro Pombeiro requested to merge pedropombeiro/455581/skip-timestamps-for-html into master

What does this MR do and why?

NOTE: This MR is best reviewed commit-by-commit.

This MR prepares the Ansi2html class to skip timestamps emitted by the runner, when the FF_TIMESTAMPS feature flag is enabled (either on the runner config itself, or as an environment variable for the job).

Logs are processed by the Gitlab::Ci::Ansi2html and Gitlab::Ci::Ansi2json classes, to produce HTML and JSON variants. Recently, the runner has introduced a FF_TIMESTAMPS feature flag, which will cause it to emit lines with a slightly modified logic:

E.g. 2024-05-14T11:19:20.000000Z 00O+Hey there!. The first 32 characters are a timestamp prefix, and must be treated as optional. If present, Ansi2html must not include it in its output.

Format:

<timestamp> <stream number><stream type><full line type>

  • timestamp: UTC RFC3339 timestamp
  • stream number: 1 byte (2 hex chars) stream number
  • stream type: E/O (Err or Out)
  • full line type: + if line is continuation of previous line, otherwise

Part of #455581 (closed)

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

image

Performance benchmarks

runs = 100; branch_name = `git branch --show-current`.strip; Benchmark.bm do |x|
  x.report("Large log with timestamps (#{branch_name})") do
    runs.times do 
      json = File.open("~/Downloads/raw-large.txt", "r") { |stream| converter = Gitlab::Ci::Ansi2html::Converter.new.convert(stream, nil) }
    end
  end
  x.report("Large log without timestamps (#{branch_name})") do
    runs.times do
      json = File.open("~/Downloads/raw-no-ts-large.txt", "r") { |stream| converter = Gitlab::Ci::Ansi2html::Converter.new.convert(stream, nil) }
    end
  end
end
Scenario user system total real
Large log with timestamps (master) 155.212844 1.004314 156.217158 159.158386
Large log without timestamps (master) 82.597942 0.550397 83.148339 84.355574
Large log with timestamps (this branch) 82.770476 0.563733 83.334209 84.738522
Large log without timestamps (this branch) 84.139308 0.487001 84.626309 86.217939

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

The tests should cover all the required scenarios.

Edited by Pedro Pombeiro

Merge request reports