high cpu usage of gitlab-monitor / puma

Summary

the gitlab-monitor process is using excessive cpu compared to the rest of the instance:

e.g. after 2 days uptime, node the total cpu usage column which is more than 10 times larger than other components

 9131 git       20   0  729652  66920   8260 S  36.1  0.1   1804:33 puma 3.12.0 (tcp://....:9101) [gitlab-monitor]                                                                                       
29546 git       20   0 8578728 1.031g  19672 S   1.7  1.6 143:56.34 sidekiq 5.2.5 gitlab-rails [0 of 25 busy]                                                                                                    
29307 git       20   0 3755864  85532  10548 S   1.3  0.1  94:52.02 /opt/gitlab/embedded/bin/gitaly /var/opt/gitlab/gitaly/config.toml                                                                           
29455 git       20   0 2559536  47680   8244 S   0.7  0.1  73:08.84 /opt/gitlab/embedded/bin/gitlab-workhorse -listenNetwork unix -listen

It is a gitlab-ee (starter) instance 11.10.2 using the omnibus package, about 190 users running on debian stretch and linux 4.19 Gitlab monitor configuration in the gitlab.rb:

################################################################################
## Gitlab monitor
##! Docs: https://docs.gitlab.com/ce/administration/monitoring/performance/prometheus.html

gitlab_monitor['enable'] = true
gitlab_monitor['listen_address'] = "..."
gitlab_monitor['listen_port'] = 9101

perf profile indicates memory allocation/garbage collection:

  10.90%  libruby.so.2.5.3    [.] match_at
   6.15%  libruby.so.2.5.3    [.] gc_sweep_step
   5.85%  libruby.so.2.5.3    [.] vm_exec_core
   3.48%  [kernel]            [k] smaps_account
   2.19%  libruby.so.2.5.3    [.] rb_enc_get_index
   2.12%  libruby.so.2.5.3    [.] vm_call_cfunc
   1.60%  libruby.so.2.5.3    [.] objspace_malloc_increase.isra.74
   1.58%  libruby.so.2.5.3    [.] rb_enc_from_index
   1.57%  libc-2.24.so        [.] malloc_usable_size

There are some errors in the monitoring log, but its frequency does not match with cpu spikes (which is 100% cpu every 1-3 seconds)

2019-05-02_12:29:21.60266 Error: No such process @ io_fillbuf - fd:22 /proc/25305/smaps
2019-05-02_12:32:19.38915 Error: No such file or directory @ rb_sysopen - /proc/25347/smaps
2019-05-02_12:32:19.47903 Error: No such file or directory @ rb_sysopen - /proc/25347/smaps
2019-05-02_12:32:19.99962 Error: No such file or directory @ rb_sysopen - /proc/25347/smaps
2019-05-02_12:39:06.39761 Error: No such file or directory @ rb_sysopen - /proc/8902/smaps
2019-05-02_12:54:05.89278 Error: No such file or directory @ rb_sysopen - /proc/19388/smaps
2019-05-02_12:54:05.89668 Error: No such file or directory @ rb_sysopen - /proc/19390/smaps
2019-05-02_12:57:35.63579 Error: No such file or directory @ rb_sysopen - /proc/30256/smaps
2019-05-02_13:08:36.45827 Error: No such file or directory @ rb_sysopen - /proc/11526/smaps
2019-05-02_13:15:36.24081 Error: No such file or directory @ rb_sysopen - /proc/2063/smaps
2019-05-02_13:28:36.43884 Error: No such file or directory @ rb_sysopen - /proc/4780/smaps
2019-05-02_13:39:06.57351 Error: No such process @ io_fillbuf - fd:25 /proc/20357/smaps
2019-05-02_13:50:51.37607 Error: No such file or directory @ rb_sysopen - /proc/13576/smaps
2019-05-02_13:52:51.39040 Error: No such process @ io_fillbuf - fd:15 /proc/24202/smaps
2019-05-02_14:11:06.04343 Error: No such file or directory @ rb_sysopen - /proc/19891/smaps
2019-05-02_14:29:05.34702 Error: No such file or directory @ rb_sysopen - /proc/28914/smaps
2019-05-02_14:33:50.76582 Error: No such file or directory @ rb_sysopen - /proc/23277/smaps
2019-05-02_14:40:32.45084 Error: No such file or directory @ rb_sysopen - /proc/12630/smaps
2019-05-02_14:40:32.58701 Error: No such file or directory @ rb_sysopen - /proc/12630/smaps

I think the issue is new since we updated from 11.8 to 11.10, but I am not confident about that.

Edited May 02, 2019 by Julian Taylor