Performance issues when processing large build traces with Ansi2html
Resources
UX @dimitrieh | FE @filipa
Single source of truth:
MVP Solution:
- Show last 50kb of log (strip the first line, so we start displaying on the first full line).
- Indicate that the log has been reverse truncated.
- Show link to download full log.
Do not load more on scroll. Do not do minimap.
Design
this issues mvp as per https://gitlab.com/gitlab-org/gitlab-ce/issues/27724#note_25268425
Other design updates have been moves to a separate issue https://gitlab.com/gitlab-org/gitlab-ce/issues/30117
Overview
When a GitLab CI runner returns a very large build trace (60MB>), we attempt to convert this using Ansi2html.convert
this causes high load and unicorn to become unresponsive.
Reproduce
Create a large build log with the required strings
cd /tmp
echo "[0;m[0KTest Line ..." >> file.txt
echo "<test<<test<<test<<test<<test<" >> file.txt
echo "<test<<test<<test<<test<<test<" >> file.txt
for i in {1..22}; do cat file.txt file.txt > file2.txt && mv file2.txt file.txt; done
Using the GDK
rails console
build = Ci::Build.first
build.trace = File.read("/tmp/file.txt");nil
build.trace.size
# => 348127232
app.get('http://localhost:3001/twitter/flight/builds/1')