Diff formatter can block a long time in Sidekiq jobs
Summary
The diff formatter using rouge
can block for a long time in Sidekiq jobs without any timeout.
Steps to reproduce
- Configure repo to send emails on push.
- Push a commit that contains a diff, triggering the hang in
rouge
. - The Sidekiq job creating the e-mail never finishes, even after 12 hours.
Example Project
waldi/test-lintian@master...test-4
This branch includes the commit that makes rouge
spin.
The web requests for the diff are killed by the hard unicorn timeout.
I'm not going to ask it to create e-mails and make Sidekiq jobs hang on gitlab.com.
What is the current bug behavior?
No e-mails on push are sent. The responsible jobs are busy running on Sidekiq. The jobs are retried after restart of Sidekiq and block the whole queue.
What is the expected correct behavior?
E-mails are sent. No job it left running for a long time.
Relevant logs and/or screenshots
Stack trace of such a blocked Job on our instance.
/srv/salsa.debian.org/gitlab/vendor/bundle/ruby/2.3.0/gems/rouge-3.1.1/lib/rouge/regex_lexer.rb:304:in `skip'
/srv/salsa.debian.org/gitlab/vendor/bundle/ruby/2.3.0/gems/rouge-3.1.1/lib/rouge/regex_lexer.rb:304:in `block in step'
/srv/salsa.debian.org/gitlab/vendor/bundle/ruby/2.3.0/gems/rouge-3.1.1/lib/rouge/regex_lexer.rb:289:in `each'
/srv/salsa.debian.org/gitlab/vendor/bundle/ruby/2.3.0/gems/rouge-3.1.1/lib/rouge/regex_lexer.rb:289:in `step'
/srv/salsa.debian.org/gitlab/vendor/bundle/ruby/2.3.0/gems/rouge-3.1.1/lib/rouge/regex_lexer.rb:270:in `stream_tokens'
/srv/salsa.debian.org/gitlab/vendor/bundle/ruby/2.3.0/gems/rouge-3.1.1/lib/rouge/lexer.rb:405:in `lex'
/srv/salsa.debian.org/gitlab/vendor/bundle/ruby/2.3.0/gems/rouge-3.1.1/lib/rouge/formatter.rb:59:in `each'
/srv/salsa.debian.org/gitlab/vendor/bundle/ruby/2.3.0/gems/rouge-3.1.1/lib/rouge/formatter.rb:59:in `token_lines'
/srv/salsa.debian.org/gitlab/lib/rouge/formatters/html_gitlab.rb:16:in `stream'
/srv/salsa.debian.org/gitlab/vendor/bundle/ruby/2.3.0/gems/rouge-3.1.1/lib/rouge/formatter.rb:37:in `format'
/srv/salsa.debian.org/gitlab/vendor/bundle/ruby/2.3.0/gems/rouge-3.1.1/lib/rouge/formatter.rb:25:in `format'
/srv/salsa.debian.org/gitlab/lib/gitlab/highlight.rb:54:in `highlight_rich'
/srv/salsa.debian.org/gitlab/lib/gitlab/highlight.rb:45:in `highlight_text'
/srv/salsa.debian.org/gitlab/lib/gitlab/highlight.rb:18:in `highlight'
/srv/salsa.debian.org/gitlab/lib/gitlab/highlight.rb:5:in `highlight'
/srv/salsa.debian.org/gitlab/lib/gitlab/diff/highlight.rb:82:in `highlighted_blob_lines'
/srv/salsa.debian.org/gitlab/lib/gitlab/diff/highlight.rb:75:in `new_lines'
/srv/salsa.debian.org/gitlab/lib/gitlab/diff/highlight.rb:53:in `highlight_line'
/srv/salsa.debian.org/gitlab/lib/gitlab/diff/highlight.rb:27:in `block in highlight'
/srv/salsa.debian.org/gitlab/lib/gitlab/diff/highlight.rb:22:in `map'
/srv/salsa.debian.org/gitlab/lib/gitlab/diff/highlight.rb:22:in `with_index'
/srv/salsa.debian.org/gitlab/lib/gitlab/diff/highlight.rb:22:in `highlight'
/srv/salsa.debian.org/gitlab/lib/gitlab/diff/file.rb:137:in `highlighted_diff_lines'
Output of checks
This bug happens on GitLab.com
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)
Related Issues and MRs
- MR for fixing perl lexer: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21263
- Issue on dev: https://dev.gitlab.org/gitlab/gitlabhq/issues/2697
- MR for adding timeout: https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2467