Job log refactor
Description
We want to add several new different features to the job log:
- Collapsible sections (done)
- Indent the content of a collapsible section
- Timestamp information
- Permanent line numbers.
Currently, we send HTML to the frontend which does not allow to have all the above in a performant way.
Solution
Update the current parser to a JSON
schema and send the JSON
to the frontend instead of HTML
:
-
backend translates the raw log into a
JSON
but has no knowledge of how it will be displayed - frontend is responsible for the behavior
Schema:
In trace.json
we will replace html
with lines
{
"lines": [
{ "offset": 10000,
"content": [ { "text": "Line A", "style": "term-fg-l-green" } ], # -> everything inside section is rendered inline
"sections": []
},
{ "offset": 10000,
"content": [ { "text": "Line B", "style": "term-fg-l-green" } ],
"sections": [ "aaa" ],
"section_header": true, # sections.last -> opening this section
},
{ "offset": 10000,
"content": [ { "text": "Line C", "style": "term-fg-l-green" } ],
"sections": [ "aaa" ]
},
{ "offset": 10000,
"content": [ { "text": "Line C-in BBB", "style": "term-fg-l-green" } ],
"sections": [ "aaa", "bbb" ] # the first line in sub-section BBB
},
{ "offset": 10000,
"content": [ { "text": "Line C-in BBB", "style": "term-fg-l-green" } ],
"sections": [ "aaa", "bbb" ] # it has all sections that this line in
},
{ "offset": 10000,
"content": [ { "text": "Line D", "style": "term-fg-l-green" } ],
"sections": [ "aaa" ], # the last line in section AAA
"section_footer": true, # we can try if we can send it?
"section_timestamp": 123.1, # execution time for sections.last
},
{ "offset": 10000,
"content": [ { "text": "Line E", "style": "term-fg-l-green" } ],
"sections": []
},
{ "offset": 10000,
"content": [
{ "text": "Uploading artifacts….",
"style": "term-fg-l-green"
},
{ "text": "id=12312321",
"style": "term-fg-l-red"
}
],
"sections": [
"aaa", "bbb", "ccc"
]
}
]
}
Incremental update
Frontend requests next incremental update providing the last offset
value.
We backend resends the last object on incremental (starting from last offset
), Frontend compares the offset
key and replaces the last element.
Backend TODOs:
-
Create new Ansi2Json class -
Extract style-related code into own class -
Implement it side-by-side with Ansi2Html and activate it via feature flag -
Detect when ANSI style does not change and append text to existing line (even if Runner sends the same ANSI sequence)
How we make backend more performant:
- by default we start to tail from 500kBs,
- send at most 100kB in chunk, if it is incomplete, send `complete:
Line numbers
For some job logs, due to size, backend will only send the last part of the log. And we do not have information about the line numbers without the rest of the log.
In order for us to be able to add a permanent link with line numbers, and because it will be consistent in the UI, we use the array's index and always start from #1
.
Timestamp
Backend will prove section_duration
with the seconds