Intermittent high CPU usage
We're using the omnibus Docker image, and the gitlab-exporter wakes up every 5 or 10 seconds or so, using 30-80% of a CPU core for a few seconds, then goes back to sleep:
This ps output shows that gitlab-exporter is the top dog in total CPU time among the GitLab processes:
estan@njord:~$ ps -u git
PID TTY TIME CMD
1808 ? 00:03:43 bundle
4872 ? 00:03:40 bundle
7317 ? 00:01:12 bundle
7594 ? 00:00:02 ruby
7599 ? 01:16:04 bundle
8647 ? 02:33:50 gitlab-exporter
11286 ? 00:11:05 gitaly
11300 ? 00:02:39 gitlab-workhors
11338 ? 00:04:17 ruby
11339 ? 00:04:20 ruby
11354 ? 00:00:26 gitaly-wrapper
20880 ? 00:03:52 bundle
23660 ? 00:03:52 bundle
26619 ? 00:04:08 bundle
27764 ? 00:07:07 bundle
29010 ? 00:03:37 bundle
31775 ? 00:03:15 bundle
estan@njord:~$
I don't believe this is https://gitlab.com/gitlab-org/gitlab-exporter/-/issues/51 since we're using GitLab 13.5.4, so using gitlab-exporter 7.1.0.
I'm not that familar with the linux perf tool, but let it run attached to the gitlab-exporter (puma) process above for a while (pid 8647), and here's the part I think is relevant:
# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 61
#
# Samples: 1K of event 'cycles:ppp'
# Event count (approx.): 2420701649125
#
# Children Self Command Shared Object Symbol
# ........ ........ ............... .................. ....................................................
#
84.82% 0.00% thread_pool.rb* libpthread-2.23.so [.] start_thread
|
---start_thread
thread_start_func_1
thread_start_func_2
thread_do_start
rb_vm_invoke_proc
rb_vm_exec
vm_exec_core
|
--84.82%--vm_call_cfunc
|
--84.76%--rb_catch_obj
vm_catch_protect
catch_i
rb_vm_exec
vm_exec_core
vm_call_cfunc
rb_catch_obj
vm_catch_protect
catch_i
rb_vm_exec
vm_exec_core
vm_call_cfunc
rb_ary_each
rb_yield
rb_vm_exec
vm_exec_core
vm_call_cfunc
rb_catch_obj
vm_catch_protect
catch_i
rb_vm_exec
vm_exec_core
vm_call_cfunc
rb_vm_call0
rb_vm_invoke_bmethod
invoke_bmethod.isra.387
rb_vm_exec
|
--84.75%--vm_exec_core
|
--84.75%--vm_call_cfunc
rb_hash_each_pair
rb_hash_foreach
rb_ensure
hash_foreach_call
|
--84.71%--each_pair_i_fast
rb_yield_values2
|
--84.70%--rb_vm_exec
vm_exec_core
vm_call_cfunc
rb_ary_each
rb_yield
rb_vm_exec
vm_exec_core
|
--84.70%--vm_call_cfunc
|
--84.70%--rb_ary_each
rb_yield
rb_vm_exec
vm_exec_core
vm_call_cfunc
|
--84.58%--rb_hash_each_pair
rb_hash_foreach
rb_ensure
hash_foreach_call
st_foreach_check
hash_foreach_iter
each_pair_i_fast
rb_yield_values2
rb_vm_exec
vm_exec_core
|
--84.58%--vm_call_cfunc
|
--84.54%--rb_thread_s_handle_interrupt
rb_yield
rb_vm_exec
vm_exec_core
vm_call_cfunc
rb_thread_s_handle_interrupt
rb_yield
rb_vm_exec
vm_exec_core
vm_call_cfunc
|
--84.54%--rb_class_s_new
rb_vm_call0
rb_vm_exec
vm_exec_core
vm_call_cfunc
|
--84.53%--enum_inject
rb_block_call
rb_iterate0
rb_vm_call0
rb_ary_each
rb_yield
vm_yield_with_cfunc.isra.365
inject_i
rb_yield_values2
rb_vm_exec
vm_exec_core
vm_call_cfunc
rb_class_s_new
rb_vm_call0
rb_vm_exec
vm_exec_core
|
--84.53%--vm_call_cfunc
|
|--29.46%--enum_group_by
| rb_block_call
| rb_iterate0
| rb_vm_call0
| |
| --29.46%--pgresult_each
| |
| |--17.41%--rb_yield
| | |
| | --17.41%--vm_yield_with_cfunc.isra.365
| | |
| | --17.40%--group_by_i
| | |
| | --10.70%--rb_hash_aset
| | |
| | --10.69%--st_update
| | |
| | --10.68%--hash_aset_str_insert
| | |
| | --10.68%--register_fstring
| | |
| | --10.68%--st_update
| |
| --12.04%--pgresult_aref
|
|--24.48%--enum_partition
| rb_block_call
| rb_iterate0
| rb_vm_call0
| |
| --18.64%--pgresult_each
| |
| --11.55%--rb_yield
| |
| --11.55%--vm_yield_with_cfunc.isra.365
| |
| --11.55%--partition_i
| |
| --11.53%--rb_yield
| |
| --10.32%--rb_vm_exec
|
--21.08%--enum_inject
rb_block_call
rb_iterate0
rb_vm_call0
|
--20.09%--pgresult_each
|
--13.43%--rb_yield
|
--13.43%--vm_yield_with_cfunc.isra.365
|
--13.43%--inject_i
rb_yield_values2
|
--12.26%--rb_vm_exec
|
--12.26%--vm_exec_core
Looks like most of the time is spent in a group_by + partition + inject?
Edited by Elvis Stansvik