Skip to content

Page rendering hangs and errors when rst (aka rest aka ReStructuredText) file is too long

Summary

When a rst file gets too long, gitlab fails to render it, the page hangs and eventually errors out.

Steps to reproduce

Example Project

https://gitlab.com/hnasarat/test/commits/master

What is the current bug behavior?

Page hangs

What is the expected correct behavior?

Renders okay or provides error explaining how to correct issue.

Relevant logs and/or screenshots

==> /var/log/gitlab/unicorn/unicorn_stderr.log <==
E, [2019-08-01T23:02:54.017651 #420] ERROR -- : worker=2 PID:26290 timeout (61s > 60s), killing
E, [2019-08-01T23:02:54.076579 #420] ERROR -- : reaped #<Process::Status: pid 26290 SIGKILL (signal 9)> worker=2
I, [2019-08-01T23:02:54.090806 #27601]  INFO -- : worker=2 ready

==> /var/log/gitlab/gitlab-workhorse/current <==
{"correlation_id":"O6CX7vqtm02","error":"badgateway: failed after 61s: EOF","level":"error","method":"GET","msg":"error","time":"2019-08-01T23:02:54Z","uri":"/editshare/devtools/blob/master/doc/hacking.rst?format=json\u0026viewer=rich"}
{"correlation_id":"O6CX7vqtm02","duration":61.134792318,"host":"gitlab.dev.editshare.com","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referer":"http://gitlab.dev.editshare.com/editshare/devtools/blob/master/doc/hacking.rst","remoteAddr":"127.0.0.1:0","remoteIp":"127.0.0.1","status":502,"system":"http","time":"2019-08-01T23:02:54Z","uri":"/editshare/devtools/blob/master/doc/hacking.rst?format=json\u0026viewer=rich","userAgent":"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0","written":2940}

When hacking.rst is being rendered, GitLab spawns a python3 script which hangs trying to write to stdout. Output from strace:

(...)
open("/opt/gitlab/embedded/lib/python3.7/site-packages/docutils/writers/html4css1/html4css1.css", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=7210, ...}) = 0
ioctl(3, TCGETS, 0x7ffdef3194d0)        = -1 ENOTTY (Inappropriate ioctl for device)
lseek(3, 0, SEEK_CUR)                   = 0
lseek(3, 0, SEEK_CUR)                   = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=7210, ...}) = 0
read(3, "/*\n:Author: David Goodger (goodg"..., 7211) = 7210
read(3, "", 1)                          = 0
close(3)                                = 0
mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3440d38000
open("/opt/gitlab/embedded/lib/python3.7/site-packages/docutils/writers/html4css1/template.txt", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=114, ...}) = 0
ioctl(3, TCGETS, 0x7ffdef319f90)        = -1 ENOTTY (Inappropriate ioctl for device)
lseek(3, 0, SEEK_CUR)                   = 0
lseek(3, 0, SEEK_CUR)                   = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=114, ...}) = 0
read(3, "%(head_prefix)s\n%(head)s\n%(style"..., 115) = 114
read(3, "", 1)                          = 0
close(3)                                = 0
write(1, "<h1 class=\"title\">Developer\342\200\231s "..., 67683

The last write ends after a minute or so with EPIPE:

write(1, "<h1 class=\"title\">Developer\342\200\231s "..., 67683) = 65536
--- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=10289, si_uid=998} ---
write(1, "1-1ubuntu2es2</code> (upstream\nv"..., 2147) = -1 EPIPE (Broken pipe)
--- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=10289, si_uid=998} ---
write(1, "1-1ubuntu2es2</code> (upstream\nv"..., 2147) = -1 EPIPE (Broken pipe)
--- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=10289, si_uid=998} ---

So It looks like a bug in some GitLab module which should actually read that output.

We noticed that when gitlab starts rendering, some python3 thing briefly appears in htop sorted by CPU usage. Then we did:

while sleep 0.1; do ps axjf | grep python3 ; done
to see what's this and noticed:
30114 31028 31027 30114 pts/2    31027 S+    1000   0:00              \_ grep python3
26753 30906  1832  1832 ?           -1 S      998   0:00  |   |       |   |   \_ python3 /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-markup-1.7.0/lib/github/commands/rest2html
30598 30964  1832  1832 ?           -1 S      998   0:00  |   |       |       \_ python3 /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-markup-1.7.0/lib/github/commands/rest2html
30114 31031 31030 30114 pts/2    31030 S+    1000   0:00              \_ grep python3
26753 30906  1832  1832 ?           -1 S      998   0:00  |   |       |   |   \_ python3 /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-markup-1.7.0/lib/github/commands/rest2html
30598 30964  1832  1832 ?           -1 S      998   0:00  |   |       |       \_ python3 /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-markup-1.7.0/lib/github/commands/rest2html
30114 31035 31034 30114 pts/2    31034 S+    1000   0:00              \_ grep python3
26753 30906  1832  1832 ?           -1 S      998   0:00  |   |       |   |   \_ python3 /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-markup-1.7.0/lib/github/commands/rest2html
30598 30964  1832  1832 ?           -1 S      998   0:00  |   |       |       \_ python3 /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-markup-1.7.0/lib/github/commands/rest2html

The last thing was to be quick enough to catch one those:

while true; do pid=$(ps axjf | grep python3 | grep commands/rest2html | awk '{print $2}'); if [[ $pid ]]; then strace -p "$pid" ; fi; done

Output of checks

This bug happens both on GitLab.com, and on my own hosted gitlab instance.

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:
Current User:	git
Using RVM:	no
Ruby Version:	2.6.3p62
Gem Version:	2.7.9
Bundler Version:1.17.3
Rake Version:	12.3.2
Redis Version:	3.2.12
Git Version:	2.21.0
Sidekiq Version:5.2.7
Go Version:	unknown


GitLab information
Version:	12.1.3
Revision:	6ccbccc2010
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	9.6.11
URL:		http://gitlab.dev.editshare.com
HTTP Clone URL:	http://gitlab.dev.editshare.com/some-group/some-project.git
SSH Clone URL:	git@gitlab.dev.editshare.com:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers:


GitLab Shell
Version:	9.3.0
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Possible fixes

IDK where in the code this is, but we investigated it at my workplace.

Edited by hash