Regular spikes in OptimizeRepository requests resulting in CPU saturation
We are having regular instances of alerts firing for saturation on Gitaly nodes, for example:
One potential cause is the surge in OptimizeRepository
, see: gitlab-com/gl-infra/production#3880 (comment 525146801)
It is suspected that the cause of the problem are scheduled maintenance jobs: gitlab-com/gl-infra/production#3156 (closed) . See slack thread here: https://gitlab.slack.com/archives/C7XJA8FSQ/p1615293649020600
There was an attempt to disable those tasks here: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/5138
Could we please get some help with confirming that the cause of these spikes are indeed maintenance tasks and with adjusting them so that they don't saturate Gitaly nodes (e.g. schedule the load differently)?