Streaming Syntax Highlighting

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Our current syntax highlighting setup is not efficient at all. For example, when we want to display a file or diff we simply load the entire thing into memory and highlight it. In case of diffs it's worse, we load/highlight both the old and new version. This can lead to large amounts of memory being consumed (as described in https://gitlab.com/gitlab-org/gitlab-ce/issues/26329).

To make this more efficient we need to be able to stream input to our syntax highlighter, e.g. one line at a time. The highlighter in turn should spit out only a single chunk/line at a time. Basically you'd end up with an interface such as:

lexer = Rouge::LexerThing.new(input_io_object_that_acts_like_an_enumerable)

while line = lexer.next_line
  # do something with "line"
end

cc @jneen