Very poor performance creating merge request, also failing at loading comments

Hi, we use Gitlab version 11.5.4-ee (62f48dfb) in a VM (kvm - configured with 12GB RAM, HDD type VirtIO raw, where we run Debian9 system x86_64, with 24 CPU cores). It runs on physical machine HP ProLiant DL580 G7 24 core, 48 thread Intel Xeon CPU E7-4807, 64GB RAM, on debian10 system. It takes about a minute to open a pull-request (merge-requst) page, and it even fails to load, that is the "comments" never fully load, and disappear. That PR has around 50 comments and around 50 commits.

During opening such PR, the server load is following (attached copy/paste from atop etc):

PRC |  sys    3.76s  |  user  23.49s  |  #proc    434  |  #zombie    0  |  #exit     13  |
CPU |  sys      37%  |  user    219%  |  irq       2%  |  idle   2125%  |  wait      3%  |
CPL |  avg1    1.16  |  avg5    1.05  |  avg15   1.16  |  csw    18247  |  intr   19984  |
MEM |  tot    11.7G  |  free    1.4G  |  cache   1.0G  |  buff  122.5M  |  slab  296.2M  |
SWP |  tot     8.0G  |  free    7.9G  |                |  vmcom   9.4G  |  vmlim  13.8G  |
LVM |  rzec--vg-var  |  busy      3%  |  read      20  |  write    140  |  avio 1.65 ms  |
LVM |  zec--vg-root  |  busy      0%  |  read       2  |  write      0  |  avio 14.0 ms  |
DSK |           vdb  |  busy      2%  |  read      19  |  write     33  |  avio 4.00 ms  |
DSK |           vda  |  busy      1%  |  read       3  |  write     44  |  avio 2.55 ms  |
NET |  transport     |  tcpi     313  |  tcpo     601  |  udpi      89  |  udpo      89  |
NET |  network       |  ipi     4703  |  ipo     4675  |  ipfrw   4246  |  deliv    406  |
NET |  ens3    ----  |  pcki    2282  |  pcko    2323  |  si 3219 Kbps  |  so  672 Kbps  |
NET |  ens10   ----  |  pcki    2238  |  pcko    2115  |  si  189 Kbps  |  so 3195 Kbps  |

   PID  THR SYSCPU USRCPU  VGROW  RGROW  RDDSK   WRDSK ST EXC S CPUNR  CPU CMD         1/5
 11881   11  0.07s  4.44s     0K -5184K     0K      4K --   - S     4  45% bundle
 18741    5  0.70s  3.70s     0K   648K     0K      0K --   - R     3  44% bundle
 20528   11  0.13s  4.14s     0K -12.4M     0K      0K --   - R     2  43% bundle
 19794   65  0.23s  3.03s     0K     0K   236K    220K --   - S    11  33% prometheus
 19093    5  0.17s  1.38s  4096K   480K     0K      0K --   - S    12  16% bundle
 19834   40  0.05s  1.21s     0K  -348K     0K      0K --   - S    10  13% bundle
 20553   11  0.20s  1.04s     0K -1832K     0K      8K --   - S    18  12% bundle
 19746    7  0.12s  0.71s     0K     0K     0K      0K --   - S     8   8% gitlab-mon
 20107   11  0.05s  0.68s     0K   200K     0K      0K --   - S     2   7% bundle
 21743    1  0.44s  0.19s     0K     0K     0K      0K --   - R     6   6% atop
   442    3  0.14s  0.37s     0K   120K     0K      0K --   - S    14   5% redis-server
 20031    1  0.00s  0.36s     0K     0K     0K      0K --   - S     1   4% postgres
 19624   56  0.09s  0.18s     0K     0K    96K      0K --   - S    15   3% gitlab-workhor
 21305    1  0.03s  0.22s  2520K  2452K     0K      0K --   - S     8   3% postgres
 21856    1  0.01s  0.18s  1908K  5864K     0K      0K --   - S    22   2% postgres
 19645   63  0.05s  0.14s     0K     0K     0K      0K --   - S    15   2% gitaly

We have this problem on all PRs, the bigger the more comments/commits there are. How to solve this? This makes Gitlab unusable for us. We can provide more info about setup if requested.

Assignee Loading
Time tracking Loading