Comparison of branches with big datasets (using data in LFS) times out (error 500)

Summary

When comparing two branches of a project with large datasets, the webui seems to time out and returns the infamous "500 Whoops, something went wrong on our end.". One of the two branches has parts of its data stored in git LFS. When trying to create a merge request "nothing happens".

Steps to reproduce

  1. Select master branch
  2. Go to the branch overview
  3. Select the branch which has data in git LFS to be compared
  4. Wait 30+ seconds
  5. The "500 Whoops, something went wrong on our end." error appears

Example Project

Not possible due to the sensitivity of the used data

What is the current bug behavior?

The "500 Whoops, something went wrong on our end." error appears

What is the expected correct behavior?

The comparison of the two branches

Relevant logs and/or screenshots

We found the following entries in /var/log/gitlab/gitlab-rails/production.log which seem to be related to the error:

Completed 500 Internal Server Error in 30833ms (ActiveRecord: 58.7ms | Elasticsearch: 0.0ms)
ActionView::Template::Error (4:Deadline Exceeded):
1: - environment = local_assigns.fetch(:environment, nil)
2: - show_whitespace_toggle = local_assigns.fetch(:show_whitespace_toggle, true)
3: - can_create_note = !@diff_notes_disabled && can?(current_user, :create_note, diffs.project)
4: - diff_files = diffs.diff_files
5: - diff_page_context = local_assigns.fetch(:diff_page_context, nil)
6:
7: .content-block.oneline-block.files-changed.diff-files-changed.js-diff-files-changed
lib/gitlab/gitaly_client/diff_stitcher.rb:15:in `each'
lib/gitlab/gitaly_client/diff_stitcher.rb:15:in `each'
lib/gitlab/git/diff_collection.rb:115:in `each_gitaly_patch'
lib/gitlab/metrics/instrumentation.rb:161:in `block in each_gitaly_patch'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `each_gitaly_patch'
lib/gitlab/git/diff_collection.rb:49:in `each'
lib/gitlab/metrics/instrumentation.rb:161:in `block in each'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `each'
lib/gitlab/git/diff_collection.rb:91:in `each_with_index'
lib/gitlab/git/diff_collection.rb:91:in `decorate!'
lib/gitlab/metrics/instrumentation.rb:161:in `block in decorate!'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `decorate!'
lib/gitlab/diff/file_collection/base.rb:34:in `diff_files'
app/views/projects/diffs/_diffs.html.haml:4:in `_app_views_projects_diffs__diffs_html_haml___1987965041244661414_70020606383200'
app/views/projects/compare/show.html.haml:9:in `_app_views_projects_compare_show_html_haml__3254105502642255833_70020603069800'
app/controllers/application_controller.rb:118:in `render'
app/controllers/projects/compare_controller.rb:28:in `show'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:450:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:444:in `set_locale'
lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:57:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/multipart.rb:117:in `call'
lib/gitlab/middleware/read_only/controller.rb:42:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

Other logs that we scanned during the branch comparison don't seem to contain any related error messages

Output of checks

Issue is happening on self-hosted GitLab server

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:		Debian 8.11
Current User:	git
Using RVM:	no
Ruby Version:	2.6.3p62
Gem Version:	2.7.9
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	3.2.12
Git Version:	2.22.0
Sidekiq Version:5.2.7
Go Version:	unknown

GitLab information
Version:	12.5.3
Revision:	225d2e5bb89
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	10.9
URL:		https://odcf-gitlab.dkfz.de
HTTP Clone URL:	https://odcf-gitlab.dkfz.de/some-group/some-project.git
SSH Clone URL:	git@odcf-gitlab.dkfz.de:some-group/some-project.git
Using LDAP:	yes
Using Omniauth:	yes
Omniauth Providers:


GitLab Shell
Version:	10.2.0
Repository storage paths:
  • local: /var/opt/gitlab/git-data/repositories
  • default: /data/gitlab/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...

Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 10.2.0 ? ... OK (10.2.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... local ... OK
default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 100 users of 100 limit.


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... no
Try fixing it:
sudo chown -R git /data/var/opt/gitlab/gitlab-rails/uploads
sudo find /data/var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} ;
sudo find /data/var/opt/gitlab/gitlab-rails/uploads -type d -not -path /data/var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} ;
For more information see:
doc/install/installation.md in section "GitLab"
Please fix the error above and rerun the checks.
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
10/4 ... yes
5/5 ... yes
6/7 ... yes
49/8 ... yes
8/9 ... yes
50/10 ... yes
48/11 ... yes
23/13 ... yes
10/15 ... yes
23/16 ... yes
23/17 ... yes
10/18 ... yes
10/20 ... yes
10/22 ... yes
10/23 ... yes
10/24 ... yes
10/25 ... yes
10/27 ... yes
10/28 ... yes
10/29 ... yes
10/30 ... yes
10/31 ... yes
10/32 ... yes
10/33 ... yes
10/34 ... yes
10/35 ... yes
10/36 ... yes
10/37 ... yes
10/38 ... yes
10/39 ... yes
10/40 ... yes
10/41 ... yes
10/42 ... yes
10/43 ... yes
10/44 ... yes
10/45 ... yes
10/46 ... yes
10/47 ... yes
10/48 ... yes
10/49 ... yes
10/50 ... yes
10/51 ... yes
10/54 ... yes
10/55 ... yes
10/56 ... yes
10/57 ... yes
10/58 ... yes
10/59 ... yes
10/61 ... yes
34/63 ... yes
34/64 ... yes
34/65 ... yes
34/66 ... yes
34/67 ... yes
34/68 ... yes
34/69 ... yes
34/70 ... yes
48/71 ... yes
48/73 ... yes
48/74 ... yes
50/75 ... yes
48/76 ... yes
5/77 ... yes
7/80 ... yes
39/81 ... yes
41/82 ... yes
5/83 ... yes
43/84 ... yes
48/85 ... yes
43/88 ... yes
51/89 ... yes
10/90 ... yes
48/91 ... yes
48/92 ... yes
48/96 ... yes
10/97 ... yes
10/98 ... yes
53/100 ... yes
53/101 ... yes
53/102 ... yes
8/103 ... yes
48/104 ... yes
48/105 ... yes
5/106 ... yes
5/107 ... yes
50/108 ... yes
8/109 ... yes
48/110 ... yes
23/111 ... yes
50/112 ... yes
12/114 ... yes
48/116 ... yes
95/117 ... yes
34/119 ... yes
24/121 ... yes
59/124 ... yes
59/125 ... yes
63/126 ... yes
23/128 ... yes
48/129 ... yes
64/130 ... yes
63/131 ... yes
63/133 ... yes
58/136 ... yes
63/137 ... yes
58/138 ... yes
10/139 ... yes
58/140 ... yes
63/141 ... yes
67/142 ... yes
72/143 ... yes
63/144 ... yes
58/145 ... yes
52/148 ... yes
58/149 ... yes
48/150 ... yes
69/152 ... yes
69/153 ... yes
34/155 ... yes
48/157 ... yes
69/158 ... yes
75/159 ... yes
35/160 ... yes
10/161 ... yes
69/162 ... yes
34/163 ... yes
63/164 ... yes
10/165 ... yes
69/166 ... yes
33/167 ... yes
69/168 ... yes
58/170 ... yes
10/171 ... yes
58/172 ... yes
23/173 ... yes
34/175 ... yes
78/176 ... yes
78/177 ... yes
78/178 ... yes
63/179 ... yes
62/180 ... yes
58/181 ... yes
58/182 ... yes
95/183 ... yes
10/184 ... yes
10/185 ... yes
10/188 ... yes
5/189 ... yes
24/190 ... yes
48/191 ... yes
10/193 ... yes
24/194 ... yes
6/195 ... yes
92/196 ... yes
61/197 ... yes
10/198 ... yes
69/199 ... yes
69/200 ... yes
94/201 ... yes
93/202 ... yes
95/203 ... yes
95/204 ... yes
95/205 ... yes
90/206 ... yes
90/207 ... yes
97/208 ... yes
7/209 ... yes
95/211 ... yes
48/212 ... yes
12/214 ... yes
40/215 ... yes
90/216 ... yes
7/217 ... yes
99/219 ... yes
44/221 ... yes
6/222 ... yes
95/223 ... yes
48/224 ... yes
70/225 ... yes
103/226 ... yes
5/227 ... yes
6/228 ... yes
6/230 ... yes
104/231 ... yes
40/232 ... yes
65/233 ... yes
65/237 ... yes
65/238 ... yes
70/239 ... yes
12/240 ... yes
65/241 ... yes
70/242 ... yes
70/243 ... yes
48/245 ... yes
69/246 ... yes
68/247 ... yes
90/248 ... yes
23/249 ... yes
10/250 ... yes
95/251 ... yes
93/252 ... yes
48/253 ... yes
70/254 ... yes
69/255 ... yes
24/256 ... yes
93/257 ... yes
116/258 ... yes
65/259 ... yes
50/260 ... yes
10/263 ... yes
115/264 ... yes
10/265 ... yes
48/266 ... yes
69/267 ... yes
70/268 ... yes
59/269 ... yes
59/270 ... yes
125/272 ... yes
124/273 ... yes
12/274 ... yes
10/275 ... yes
10/276 ... yes
48/277 ... yes
48/278 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.3)
Git version >= 2.22.0 ? ... yes (2.22.0)
Git user has default SSH configuration? ... yes
Active users: ... 92
Is authorized keys file accessible? ... yes


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

Unknown

Edited by 🤖 GitLab Bot 🤖