502 timeout error when comparing commits with changes from many languages
### Summary
When trying to compare two commits (either through /compare or a merge request) GitLab will report a 502 error. I am unsure the exact cause, but I have been able to reproduce it on gitlab.com with just two commits. The commit contains just a few XML changes to some UTF-8 XML files.
Here is the query which will cause the 502 error.
https://gitlab.com/dkruger/gitlab-compare-502/compare/e0692c7e5247215270b8c293d97e781a488bcdb7...847229f3e074657dffb516343b2b6ccc15ff637a
Update:
I believe it is related to having multiple languages in one commit. I did some more experimenting, and if I split each files changes out into their own commit, then the comparison works fine. I get the correct diff. You can see this with the de/es/fr/it/ja/pt/zh branches on the repo I have hosted on gitlab.com.
### Steps to reproduce
1. Got to my repo hosted on gitlab.com https://gitlab.com/dkruger/gitlab-compare-502
2. Compare the first commit to the second commit, or just click on this link https://gitlab.com/dkruger/gitlab-compare-502/compare/e0692c7e5247215270b8c293d97e781a488bcdb7...847229f3e074657dffb516343b2b6ccc15ff637a
### Expected behavior
I'd expect to see a very small diff of the ts files which are just UTF-8 encoded XML files.
### Actual behavior
GitLab process for a long while and eventually just returns a 502
### Relevant logs and/or screenshots
Here's the gitlab-workhorse error from our server
```
2016-09-19_16:05:54.27900 2016/09/19 12:05:54 error: proxyRoundTripper: GET "/david.kruger/test-bad-commit/compare/e0692c7e5247215270b8c293d97e781a488bcdb7...master" failed with: "EOF"
2016-09-19_16:05:54.27902 2016/09/19 12:05:54 ErrorPage: serving predefined error page: 502
2016-09-19_16:05:54.27918 gitlab.ds.jdsu.net @ - - [2016-09-19 12:04:53.982869679 -0400 EDT] "GET /david.kruger/test-bad-commit/compare/e0692c7e5247215270b8c293d97e781a488bcdb7...master HTTP/1.1" 502 2422 "http://gitlab.ds.jdsu.net/david.kruger/test-bad-commit/compare?from=master&to=master" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36" 60.296169
```
and unicorn_stderr.log showing the timeout
```
E, [2016-09-19T12:05:54.242341 #2522] ERROR -- : worker=4 PID:29618 timeout (61s > 60s), killing
E, [2016-09-19T12:05:54.279171 #2522] ERROR -- : reaped #<Process::Status: pid 29618 SIGKILL (signal 9)> worker=4
```
I don't see any other error message in the log to indicate what the issue is.
### Output of checks
#### Results of GitLab application Check
```
root@gitlab-1:/var/log/gitlab# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab Shell ...
GitLab Shell version >= 3.4.0 ? ... OK (3.4.0)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
109/33 ... ok
88/38 ... ok
117/39 ... ok
117/40 ... ok
117/41 ... ok
117/42 ... ok
117/43 ... ok
117/44 ... ok
117/46 ... ok
117/48 ... ok
117/49 ... ok
117/51 ... ok
117/52 ... ok
117/53 ... ok
117/54 ... ok
117/55 ... ok
117/57 ... ok
117/58 ... ok
117/59 ... ok
117/60 ... ok
117/61 ... ok
117/64 ... ok
117/65 ... ok
117/66 ... ok
117/67 ... ok
117/68 ... ok
117/69 ... ok
117/70 ... ok
117/71 ... ok
117/73 ... ok
117/74 ... ok
117/77 ... ok
117/98 ... ok
109/122 ... ok
109/123 ... ok
109/124 ... ok
117/128 ... ok
109/132 ... ok
109/133 ... ok
3/135 ... ok
106/137 ... ok
89/140 ... ok
106/142 ... ok
106/143 ... ok
88/145 ... ok
3/148 ... ok
3/152 ... ok
40/153 ... ok
89/156 ... ok
79/157 ... ok
88/158 ... ok
89/160 ... ok
99/161 ... ok
98/164 ... ok
5/168 ... repository is empty
1/170 ... repository is empty
3/171 ... ok
88/172 ... ok
44/173 ... ok
106/175 ... ok
88/177 ... ok
65/178 ... ok
106/181 ... ok
107/182 ... ok
107/183 ... ok
107/184 ... ok
107/185 ... ok
107/186 ... ok
88/187 ... ok
88/188 ... ok
3/205 ... ok
3/234 ... ok
44/243 ... ok
53/244 ... ok
90/245 ... ok
3/247 ... ok
3/251 ... ok
3/252 ... ok
88/253 ... ok
3/256 ... ok
109/260 ... ok
116/261 ... ok
116/262 ... ok
116/263 ... ok
116/264 ... ok
116/265 ... ok
116/266 ... ok
116/267 ... ok
106/268 ... ok
37/269 ... ok
90/271 ... ok
116/272 ... ok
3/273 ... ok
3/275 ... ok
126/278 ... ok
126/279 ... ok
126/280 ... ok
1/281 ... repository is empty
1/282 ... repository is empty
1/283 ... repository is empty
1/284 ... repository is empty
1/285 ... repository is empty
1/286 ... repository is empty
1/287 ... repository is empty
1/288 ... repository is empty
1/289 ... repository is empty
1/290 ... repository is empty
1/291 ... repository is empty
1/292 ... repository is empty
1/293 ... repository is empty
1/294 ... repository is empty
1/295 ... repository is empty
1/296 ... repository is empty
1/297 ... repository is empty
1/298 ... repository is empty
1/299 ... repository is empty
1/300 ... repository is empty
1/301 ... repository is empty
1/302 ... repository is empty
1/303 ... repository is empty
1/304 ... repository is empty
1/305 ... repository is empty
1/306 ... repository is empty
1/307 ... repository is empty
1/308 ... repository is empty
1/309 ... repository is empty
1/310 ... repository is empty
1/311 ... repository is empty
1/312 ... repository is empty
1/313 ... repository is empty
88/314 ... ok
106/316 ... ok
98/319 ... ok
98/320 ... ok
109/322 ... ok
97/323 ... ok
44/324 ... ok
109/325 ... ok
109/326 ... ok
109/327 ... ok
106/328 ... ok
3/330 ... ok
106/331 ... ok
137/333 ... ok
106/334 ... ok
99/341 ... ok
3/342 ... ok
132/343 ... ok
107/344 ... ok
44/345 ... ok
88/347 ... ok
99/356 ... ok
88/357 ... ok
3/358 ... ok
137/359 ... ok
118/360 ... repository is empty
3/361 ... ok
149/363 ... ok
137/365 ... ok
88/366 ... ok
109/368 ... ok
154/371 ... ok
157/372 ... ok
106/374 ... ok
3/375 ... ok
88/376 ... ok
149/377 ... ok
109/378 ... ok
44/379 ... ok
40/380 ... repository is empty
157/382 ... ok
116/383 ... ok
37/384 ... ok
88/386 ... ok
3/387 ... ok
109/389 ... ok
109/390 ... ok
106/391 ... ok
44/392 ... ok
178/393 ... ok
177/395 ... ok
187/396 ... ok
109/399 ... ok
109/400 ... ok
109/401 ... ok
88/408 ... ok
187/412 ... ok
187/413 ... ok
40/414 ... ok
226/415 ... ok
187/416 ... ok
142/417 ... ok
226/418 ... ok
179/419 ... ok
44/420 ... ok
187/421 ... ok
3/424 ... ok
3/425 ... ok
193/426 ... ok
149/427 ... repository is empty
109/428 ... ok
226/429 ... ok
99/430 ... ok
226/432 ... ok
36/433 ... ok
226/434 ... ok
36/435 ... repository is empty
226/436 ... ok
194/437 ... repository is empty
201/439 ... ok
201/440 ... ok
88/441 ... ok
226/442 ... ok
201/444 ... ok
74/445 ... ok
198/446 ... ok
174/447 ... ok
3/448 ... ok
187/449 ... ok
149/450 ... ok
109/451 ... ok
159/452 ... ok
159/453 ... ok
187/454 ... ok
180/455 ... ok
187/457 ... ok
159/459 ... ok
159/461 ... ok
226/462 ... ok
187/463 ... ok
44/465 ... ok
88/466 ... ok
180/467 ... ok
209/468 ... ok
220/469 ... ok
47/470 ... ok
109/471 ... ok
40/473 ... ok
217/474 ... ok
65/475 ... ok
138/476 ... ok
3/477 ... ok
3/479 ... ok
109/480 ... ok
47/481 ... ok
47/482 ... ok
109/483 ... ok
47/484 ... ok
187/485 ... ok
109/486 ... ok
109/487 ... ok
109/488 ... ok
226/489 ... ok
226/490 ... ok
232/492 ... ok
36/493 ... ok
109/494 ... ok
138/495 ... ok
227/496 ... ok
44/497 ... ok
44/498 ... ok
3/499 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
LDAP users with access to your GitLab server (only showing the first 100 results)
Server: ldapmain
<<REDACTED>>
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
109/33 ... yes
88/38 ... yes
117/39 ... yes
117/40 ... yes
117/41 ... yes
117/42 ... yes
117/43 ... yes
117/44 ... yes
117/46 ... yes
117/48 ... yes
117/49 ... yes
117/51 ... yes
117/52 ... yes
117/53 ... yes
117/54 ... yes
117/55 ... yes
117/57 ... yes
117/58 ... yes
117/59 ... yes
117/60 ... yes
117/61 ... yes
117/64 ... yes
117/65 ... yes
117/66 ... yes
117/67 ... yes
117/68 ... yes
117/69 ... yes
117/70 ... yes
117/71 ... yes
117/73 ... yes
117/74 ... yes
117/77 ... yes
117/98 ... yes
109/122 ... yes
109/123 ... yes
109/124 ... yes
117/128 ... yes
109/132 ... yes
109/133 ... yes
3/135 ... yes
106/137 ... yes
89/140 ... yes
106/142 ... yes
106/143 ... yes
88/145 ... yes
3/148 ... yes
3/152 ... yes
40/153 ... yes
89/156 ... yes
79/157 ... yes
88/158 ... yes
89/160 ... yes
99/161 ... yes
98/164 ... yes
5/168 ... yes
1/170 ... yes
3/171 ... yes
88/172 ... yes
44/173 ... yes
106/175 ... yes
88/177 ... yes
65/178 ... yes
106/181 ... yes
107/182 ... yes
107/183 ... yes
107/184 ... yes
107/185 ... yes
107/186 ... yes
88/187 ... yes
88/188 ... yes
3/205 ... yes
3/234 ... yes
44/243 ... yes
53/244 ... yes
90/245 ... yes
3/247 ... yes
3/251 ... yes
3/252 ... yes
88/253 ... yes
3/256 ... yes
109/260 ... yes
116/261 ... yes
116/262 ... yes
116/263 ... yes
116/264 ... yes
116/265 ... yes
116/266 ... yes
116/267 ... yes
106/268 ... yes
37/269 ... yes
90/271 ... yes
116/272 ... yes
3/273 ... yes
3/275 ... yes
126/278 ... yes
126/279 ... yes
126/280 ... yes
1/281 ... yes
1/282 ... yes
1/283 ... yes
1/284 ... yes
1/285 ... yes
1/286 ... yes
1/287 ... yes
1/288 ... yes
1/289 ... yes
1/290 ... yes
1/291 ... yes
1/292 ... yes
1/293 ... yes
1/294 ... yes
1/295 ... yes
1/296 ... yes
1/297 ... yes
1/298 ... yes
1/299 ... yes
1/300 ... yes
1/301 ... yes
1/302 ... yes
1/303 ... yes
1/304 ... yes
1/305 ... yes
1/306 ... yes
1/307 ... yes
1/308 ... yes
1/309 ... yes
1/310 ... yes
1/311 ... yes
1/312 ... yes
1/313 ... yes
88/314 ... yes
106/316 ... yes
98/319 ... yes
98/320 ... yes
109/322 ... yes
97/323 ... yes
44/324 ... yes
109/325 ... yes
109/326 ... yes
109/327 ... yes
106/328 ... yes
3/330 ... yes
106/331 ... yes
137/333 ... yes
106/334 ... yes
99/341 ... yes
3/342 ... yes
132/343 ... yes
107/344 ... yes
44/345 ... yes
88/347 ... yes
99/356 ... yes
88/357 ... yes
3/358 ... yes
137/359 ... yes
118/360 ... yes
3/361 ... yes
149/363 ... yes
137/365 ... yes
88/366 ... yes
109/368 ... yes
154/371 ... yes
157/372 ... yes
106/374 ... yes
3/375 ... yes
88/376 ... yes
149/377 ... yes
109/378 ... yes
44/379 ... yes
40/380 ... yes
157/382 ... yes
116/383 ... yes
37/384 ... yes
88/386 ... yes
3/387 ... yes
109/389 ... yes
109/390 ... yes
106/391 ... yes
44/392 ... yes
178/393 ... yes
177/395 ... yes
187/396 ... yes
109/399 ... yes
109/400 ... yes
109/401 ... yes
88/408 ... yes
187/412 ... yes
187/413 ... yes
40/414 ... yes
226/415 ... yes
187/416 ... yes
142/417 ... yes
226/418 ... yes
179/419 ... yes
44/420 ... yes
187/421 ... yes
3/424 ... yes
3/425 ... yes
193/426 ... yes
149/427 ... yes
109/428 ... yes
226/429 ... yes
99/430 ... yes
226/432 ... yes
36/433 ... yes
226/434 ... yes
36/435 ... yes
226/436 ... yes
194/437 ... yes
201/439 ... yes
201/440 ... yes
88/441 ... yes
226/442 ... yes
201/444 ... yes
74/445 ... yes
198/446 ... yes
174/447 ... yes
3/448 ... yes
187/449 ... yes
149/450 ... yes
109/451 ... yes
159/452 ... yes
159/453 ... yes
187/454 ... yes
180/455 ... yes
187/457 ... yes
159/459 ... yes
159/461 ... yes
226/462 ... yes
187/463 ... yes
44/465 ... yes
88/466 ... yes
180/467 ... yes
209/468 ... yes
220/469 ... yes
47/470 ... yes
109/471 ... yes
40/473 ... yes
217/474 ... yes
65/475 ... yes
138/476 ... yes
3/477 ... yes
3/479 ... yes
109/480 ... yes
47/481 ... yes
47/482 ... yes
109/483 ... yes
47/484 ... yes
187/485 ... yes
109/486 ... yes
109/487 ... yes
109/488 ... yes
226/489 ... yes
226/490 ... yes
232/492 ... yes
36/493 ... yes
109/494 ... yes
138/495 ... yes
227/496 ... yes
44/497 ... yes
44/498 ... yes
3/499 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.1)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.7.4)
Active users: 199
Checking GitLab ... Finished
```
#### Results of GitLab environment info
```
root@gitlab-1:/var/log/gitlab# sudo gitlab-rake gitlab:env:info
System information
System: Ubuntu 14.04
Current User: git
Using RVM: no
Ruby Version: 2.3.1p112
Gem Version: 2.6.6
Bundler Version:2.3.0
Rake Version: 10.5.0
Sidekiq Version:4.1.4
GitLab information
Version: 8.11.6
Revision: 410d751
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL: http://gitlab.ds.jdsu.net
HTTP Clone URL: http://gitlab.ds.jdsu.net/some-group/some-project.git
SSH Clone URL: git@gitlab.ds.jdsu.net:some-group/some-project.git
Using LDAP: yes
Using Omniauth: no
GitLab Shell
Version: 3.4.0
Repository storage paths:
- default: /mnt/atlantic/gitlab-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git: /opt/gitlab/embedded/bin/git
```
### Possible fixes
No clue, sorry.
issue