Very poor performance creating merge request, also failing at loading comments
Hi, we use Gitlab version 11.5.4-ee (62f48dfb) in a VM (kvm - configured with 12GB RAM, HDD type VirtIO raw, where we run Debian9 system x86_64, with 24 CPU cores). It runs on physical machine HP ProLiant DL580 G7 24 core, 48 thread Intel Xeon CPU E7-4807, 64GB RAM, on debian10 system. It takes about a minute to open a pull-request (merge-requst) page, and it even fails to load, that is the "comments" never fully load, and disappear. That PR has around 50 comments and around 50 commits.
During opening such PR, the server load is following (attached copy/paste from atop etc):
PRC | sys 3.76s | user 23.49s | #proc 434 | #zombie 0 | #exit 13 |
CPU | sys 37% | user 219% | irq 2% | idle 2125% | wait 3% |
CPL | avg1 1.16 | avg5 1.05 | avg15 1.16 | csw 18247 | intr 19984 |
MEM | tot 11.7G | free 1.4G | cache 1.0G | buff 122.5M | slab 296.2M |
SWP | tot 8.0G | free 7.9G | | vmcom 9.4G | vmlim 13.8G |
LVM | rzec--vg-var | busy 3% | read 20 | write 140 | avio 1.65 ms |
LVM | zec--vg-root | busy 0% | read 2 | write 0 | avio 14.0 ms |
DSK | vdb | busy 2% | read 19 | write 33 | avio 4.00 ms |
DSK | vda | busy 1% | read 3 | write 44 | avio 2.55 ms |
NET | transport | tcpi 313 | tcpo 601 | udpi 89 | udpo 89 |
NET | network | ipi 4703 | ipo 4675 | ipfrw 4246 | deliv 406 |
NET | ens3 ---- | pcki 2282 | pcko 2323 | si 3219 Kbps | so 672 Kbps |
NET | ens10 ---- | pcki 2238 | pcko 2115 | si 189 Kbps | so 3195 Kbps |
PID THR SYSCPU USRCPU VGROW RGROW RDDSK WRDSK ST EXC S CPUNR CPU CMD 1/5
11881 11 0.07s 4.44s 0K -5184K 0K 4K -- - S 4 45% bundle
18741 5 0.70s 3.70s 0K 648K 0K 0K -- - R 3 44% bundle
20528 11 0.13s 4.14s 0K -12.4M 0K 0K -- - R 2 43% bundle
19794 65 0.23s 3.03s 0K 0K 236K 220K -- - S 11 33% prometheus
19093 5 0.17s 1.38s 4096K 480K 0K 0K -- - S 12 16% bundle
19834 40 0.05s 1.21s 0K -348K 0K 0K -- - S 10 13% bundle
20553 11 0.20s 1.04s 0K -1832K 0K 8K -- - S 18 12% bundle
19746 7 0.12s 0.71s 0K 0K 0K 0K -- - S 8 8% gitlab-mon
20107 11 0.05s 0.68s 0K 200K 0K 0K -- - S 2 7% bundle
21743 1 0.44s 0.19s 0K 0K 0K 0K -- - R 6 6% atop
442 3 0.14s 0.37s 0K 120K 0K 0K -- - S 14 5% redis-server
20031 1 0.00s 0.36s 0K 0K 0K 0K -- - S 1 4% postgres
19624 56 0.09s 0.18s 0K 0K 96K 0K -- - S 15 3% gitlab-workhor
21305 1 0.03s 0.22s 2520K 2452K 0K 0K -- - S 8 3% postgres
21856 1 0.01s 0.18s 1908K 5864K 0K 0K -- - S 22 2% postgres
19645 63 0.05s 0.14s 0K 0K 0K 0K -- - S 15 2% gitaly
We have this problem on all PRs, the bigger the more comments/commits there are. How to solve this? This makes Gitlab unusable for us. We can provide more info about setup if requested.