High CPU usage caused by ruby (gitaly-ruby) processes

Summary

From the very begging we have performance issues with Gitlab installation. CPU usage by gitaly-ruby processes is very high which has an impact on performance. Most of the time UI is very slow (load of repository tree takes up to 50 seconds or more) and pull or push requests are timing out.

Steps to reproduce

Use the Gitlab Web UI

What is the current bug behavior?

High CPU Load, very slow Web UI

What is the expected correct behavior?

CPU is not overloaded.

Relevant logs and/or screenshots

strace -p 53109
strace: Process 53109 attached
futex(0x5587f790bf88, FUTEX_WAIT_PRIVATE, 0, NULL^Cstrace: Process 53109 detached
 <detached ...>

info threads for random ruby (gitlay-ruby) process

─── Threads ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
[41] id 53595 name grpc_global_tim from 0x00007f263454bed9 in futex_reltimed_wait_cancelable+29 at ../sysdeps/unix/sysv/linux/futex-internal.h:142
[40] id 53152 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[39] id 53151 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[38] id 53150 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[37] id 53149 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[36] id 53148 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[35] id 53147 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[34] id 53146 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[33] id 53145 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[32] id 53144 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[31] id 53143 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[30] id 53142 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[29] id 53141 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[28] id 53140 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[27] id 53139 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[26] id 53138 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[25] id 53137 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[24] id 53136 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[23] id 53135 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[22] id 53134 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[21] id 53133 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[20] id 53132 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[19] id 53131 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[18] id 53130 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[17] id 53129 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[16] id 53128 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[15] id 53127 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[14] id 53126 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[13] id 53125 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[12] id 53124 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[11] id 53123 name rpc_server.rb:* from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[10] id 53122 name gitaly-ruby:51 from 0x00007f263487ebb7 in epoll_wait+87 at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
[9] id 53121 name gitaly-ruby:62 from 0x00007f263454bed9 in futex_reltimed_wait_cancelable+29 at ../sysdeps/unix/sysv/linux/futex-internal.h:142
[8] id 53120 name gitaly-ruby:41 from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[7] id 53119 name ruby from 0x00007f263487ebb7 in epoll_wait+87 at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
[6] id 53118 name ruby from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[5] id 53117 name grpc_global_tim from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[4] id 53116 name resolver-execut from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[3] id 53115 name default-executo from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
[2] id 53113 name ruby-timer-thr from 0x00007f2634871bf9 in __GI___poll+73 at ../sysdeps/unix/sysv/linux/poll.c:29
[1] id 53109 name ruby from 0x00007f263454b9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
0x00007f263454b9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x5587f790bf88) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88	../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.

Results of GitLab environment info

Expand for output related to GitLab environment info

System information System: Ubuntu 18.04 Current User: git Using RVM: no Ruby Version: 2.5.3p105 Gem Version: 2.7.6 Bundler Version:1.16.6 Rake Version: 12.3.1 Redis Version: 3.2.12 Git Version: 2.18.1 Sidekiq Version:5.2.3 Go Version: go1.10.4 linux/amd64

GitLab information Version: 11.6.0 Revision: 075f96092a7 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql Using LDAP: yes Using Omniauth: yes Omniauth Providers:

GitLab Shell Version: 8.4.3 Repository storage paths:

default: /data/gitlab/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Passing

Edited Dec 26, 2018 by miko