Performance issues when processing large build traces with Ansi2html

Resources

UX @dimitrieh | FE @filipa

Single source of truth:

MVP Solution:

  1. Show last 50kb of log (strip the first line, so we start displaying on the first full line).
  2. Indicate that the log has been reverse truncated.
  3. Show link to download full log.

Do not load more on scroll. Do not do minimap.

Design

this issues mvp as per https://gitlab.com/gitlab-org/gitlab-ce/issues/27724#note_25268425

image

Other design updates have been moves to a separate issue https://gitlab.com/gitlab-org/gitlab-ce/issues/30117

Overview

When a GitLab CI runner returns a very large build trace (60MB>), we attempt to convert this using Ansi2html.convert this causes high load and unicorn to become unresponsive.

Reproduce

Create a large build log with the required strings

cd /tmp
echo "[0;m[0KTest Line ..." >> file.txt
echo "<test<<test<<test<<test<<test<" >> file.txt
echo "<test<<test<<test<<test<<test<" >> file.txt
for i in {1..22}; do cat file.txt file.txt > file2.txt && mv file2.txt file.txt; done

Using the GDK

rails console
build = Ci::Build.first
build.trace = File.read("/tmp/file.txt");nil
build.trace.size
# => 348127232
app.get('http://localhost:3001/twitter/flight/builds/1')

Links