issue links endpoint is very slow

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Summary

We noticed a major slowdown in our gitlab instance.

We are running GitLab EE Starter on GCP with LDAP enabled.

We traced the problem back to the GET /projects/:id/issues/:issue_iid/links endpoint.

Things I observed on our instance:

• Said endpoint takes between 2 and 30 seconds. Sometimes it even times out.
• Response times get drastically worse the more issues are linked
• Response time for other endpoints is around 200ms
• I don't notice any slowdown using the UI instead of the API. The links load pretty fast

Steps to reproduce

1. Have a issue with 20+ linked issues
2. Get the issue links using GET /projects/:id/issues/:issue_iid/links

Example Project

The issue does not seem to be as noticeable on gitlab.com as on our instance. gitlab.com might just tank it. We run our instance on a n1-standard-2 GCP instance.

• https://gitlab.com/api/v4/projects/12192511/issues/296/links takes +2 seconds
• while other endpoints like https://gitlab.com/api/v4/projects/12192511/issues/296 just take arround 250ms (wow, gitlab.com is fast now 😄 )

Relevant logs and/or screenshots

I'm not good at reading the profiling results, but i think that there might be an issue with auth checking? It looks like LDAP auth checking takes a lot of time.

This could also explain why gitlab.com does not seem to be affected that much.

Profile of a example request 1 (took ~14seconds)

Profile of a example request 2 (timed out)

Output of checks

This bug happens on GitLab.com (but is less noticeable)

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:         Ubuntu 16.04
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.5.3p105
Gem Version:    2.7.6
Bundler Version:1.17.3
Rake Version:   12.3.2
Redis Version:  3.2.12
Git Version:    2.18.1
Sidekiq Version:5.2.5
Go Version:     unknown
GitLab information
Version:        11.10.4-ee
Revision:       88a3c791734
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     9.6.11
URL:            https://gitlab.hydra-newmedia.cloud
HTTP Clone URL: https://gitlab.hydra-newmedia.cloud/some-group/some-project.git
SSH Clone URL:  git@gitlab.hydra-newmedia.cloud:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers: gitlab
GitLab Shell
Version:        9.0.0
Repository storage paths:

• default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 9.0.0 ? ... OK (9.0.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Checking Reply by email ...
IMAP server credentials are correct? ... yes
Init.d configured correctly? ... skipped
MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
DN: cn=Max Mustermann,ou=benutzer,dc=hydra,dc=local	 sAMAccountName: mmuster
DN: cn=test,ou=benutzer,dc=hydra,dc=local	 sAMAccountName: test
[omitted for privacy]
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
12/2 ... yes
12/3 ... yes
5/5 ... yes
3/7 ... yes
3/9 ... yes
3/12 ... yes
3/13 ... yes
12/14 ... yes
5/15 ... yes
3/17 ... yes
12/18 ... yes
5/19 ... yes
3/20 ... yes
12/21 ... yes
3/23 ... yes
3/24 ... yes
3/26 ... yes
12/29 ... yes
12/30 ... yes
12/31 ... yes
12/32 ... yes
12/33 ... yes
3/34 ... yes
5/35 ... yes
3/36 ... yes
3/37 ... yes
5/38 ... yes
1/39 ... yes
7/40 ... yes
7/41 ... yes
7/42 ... yes
7/43 ... yes
7/44 ... yes
7/45 ... yes
7/46 ... yes
7/48 ... yes
7/49 ... yes
7/50 ... yes
7/51 ... yes
7/52 ... yes
7/53 ... yes
7/54 ... yes
7/55 ... yes
7/56 ... yes
7/57 ... yes
7/58 ... yes
7/59 ... yes
7/60 ... yes
7/61 ... yes
7/62 ... yes
7/63 ... yes
7/64 ... yes
7/65 ... yes
7/66 ... yes
7/67 ... yes
19/68 ... yes
7/69 ... yes
3/70 ... yes
5/77 ... yes
5/80 ... yes
5/82 ... yes
5/83 ... yes
3/84 ... yes
12/85 ... yes
7/86 ... yes
5/87 ... yes
3/90 ... yes
14/92 ... yes
3/93 ... yes
3/94 ... yes
3/95 ... yes
3/96 ... yes
3/97 ... yes
3/98 ... yes
3/99 ... yes
3/100 ... yes
3/102 ... yes
3/103 ... yes
3/104 ... yes
3/105 ... yes
3/106 ... yes
3/107 ... yes
14/108 ... yes
3/109 ... yes
3/110 ... yes
3/111 ... yes
3/112 ... yes
3/113 ... yes
3/114 ... yes
3/115 ... yes
3/116 ... yes
3/117 ... yes
3/118 ... yes
3/119 ... yes
3/120 ... yes
3/121 ... yes
3/122 ... yes
3/123 ... yes
3/124 ... yes
3/125 ... yes
3/126 ... yes
3/127 ... yes
3/128 ... yes
3/129 ... yes
3/130 ... yes
3/131 ... yes
12/132 ... yes
12/133 ... yes
14/134 ... yes
7/135 ... yes
7/136 ... yes
3/137 ... yes
5/138 ... yes
3/140 ... yes
5/142 ... yes
5/143 ... yes
5/144 ... yes
5/145 ... yes
12/146 ... yes
3/147 ... yes
3/149 ... yes
5/150 ... yes
3/151 ... yes
3/152 ... yes
3/153 ... yes
5/157 ... yes
5/159 ... yes
3/160 ... yes
5/161 ... yes
3/162 ... yes
3/163 ... yes
3/164 ... yes
5/167 ... yes
3/168 ... yes
19/169 ... yes
34/170 ... yes
3/171 ... yes
5/174 ... yes
3/175 ... yes
5/176 ... yes
3/177 ... yes
3/178 ... yes
5/179 ... yes
12/180 ... yes
3/181 ... yes
12/182 ... yes
3/183 ... yes
3/184 ... yes
1/185 ... yes
3/186 ... yes
5/187 ... yes
5/189 ... yes
3/190 ... yes
3/191 ... yes
3/192 ... yes
78/193 ... yes
3/194 ... yes
39/195 ... yes
3/196 ... yes
3/198 ... yes
48/199 ... yes
25/200 ... yes
3/201 ... yes
3/204 ... yes
3/205 ... yes
3/207 ... yes
3/208 ... yes
3/209 ... yes
3/210 ... yes
3/211 ... yes
54/212 ... yes
3/213 ... yes
3/214 ... yes
5/215 ... yes
3/217 ... yes
3/219 ... yes
31/220 ... yes
54/221 ... yes
3/222 ... yes
3/223 ... yes
3/224 ... yes
3/225 ... yes
31/226 ... yes
3/227 ... yes
3/228 ... yes
3/229 ... yes
3/230 ... yes
54/231 ... yes
57/232 ... yes
57/233 ... yes
12/234 ... yes
3/235 ... yes
40/236 ... yes
57/237 ... yes
54/238 ... yes
3/239 ... yes
3/240 ... yes
57/242 ... yes
3/243 ... yes
5/244 ... yes
54/245 ... yes
77/253 ... yes
12/254 ... yes
57/256 ... yes
3/257 ... yes
3/258 ... yes
5/259 ... yes
1/260 ... yes
3/261 ... yes
3/263 ... yes
54/264 ... yes
19/265 ... yes
3/266 ... yes
3/267 ... yes
5/268 ... yes
3/270 ... yes
3/271 ... yes
3/272 ... yes
21/273 ... yes
54/274 ... yes
5/275 ... yes
5/276 ... yes
3/277 ... yes
40/278 ... yes
3/279 ... yes
3/280 ... yes
3/281 ... yes
3/283 ... yes
80/284 ... yes
3/285 ... yes
3/287 ... yes
80/288 ... yes
80/289 ... yes
3/290 ... yes
3/293 ... yes
3/295 ... yes
15/296 ... yes
5/297 ... yes
3/298 ... yes
3/300 ... yes
3/301 ... yes
3/302 ... yes
84/303 ... yes
37/304 ... yes
26/305 ... yes
3/306 ... yes
3/308 ... yes
3/309 ... yes
3/310 ... yes
3/312 ... yes
3/314 ... yes
1/315 ... yes
3/318 ... yes
3/319 ... yes
3/320 ... yes
3/321 ... yes
3/322 ... yes
1/323 ... yes
3/324 ... yes
3/325 ... yes
12/326 ... yes
29/327 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.5.3)
Git version >= 2.18.0 ? ... yes (2.18.1)
Git user has default SSH configuration? ... yes
Active users: ... 46
Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes

We would be fine with an endpoint that just returns the linked ids as numbers. We don't need all issues. Don't know if that would improved performance though.