Sidekiq crashes with sd_markdown_render error when under high load
Sidekiq crashes almost daily with this error:
2017-08-17T16:26:58.515Z 4084 TID-ox1fzyws8 ActionMailer::DeliveryJob JID-f2593e2cebd93caba716c59c INFO: done: 5.978 sec
itlab [23 of 25 busy]: markdown.c:2897: sd_markdown_render: Assertion `md->work_bufs[0].size == 0' failed.It would appear this occurs when sidekiq is under a fairly high load, with Sidekiq hitting 20+. Restarting gitlab resolves the issue until the next time. Note that after restarting Gitlab Sidekiq caps out at 25 until the queue is empty again.
Expand for output related to GitLab environment info
System information System: Ubuntu 14.04 Current User: git Using RVM: no Ruby Version: 2.3.3p222 Gem Version: 2.5.2 Bundler Version:1.15.3 Rake Version: 10.5.0 Redis Version: 2.8.4 Git Version: 2.7.3 Sidekiq Version:5.0.0 Go Version: go1.8.3 linux/amd64
GitLab information Version: 9.4.4 Revision: 2c79d92 Directory: /home/git/gitlab DB Adapter: postgresql URL: https://gitlab.domain.com HTTP Clone URL: https://gitlab.domain.com/some-group/some-project.git SSH Clone URL: [email protected]:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: khronos
GitLab Shell Version: 5.3.1 Repository storage paths:
- default: /home/git/repositories Hooks: /home/git/gitlab-shell/hooks Git: /usr/local/bin/git
Possible fixes
This error is possibly related to redcarpet and these issues: https://github.com/vmg/redcarpet/issues/570 https://github.com/vmg/redcarpet/issues/176