High Gitaly CPU usage/load average causing issues

Summary

As of 10.4.0 update (from 10.1) Gitaly CPU usage is out of control to the point that the web UI is slow, pulls and pushes are timing out, etc.

Steps to reproduce

Run GitLab with thousands of users and projects via SAN storage.

What is the current bug behavior?

What is the expected correct behavior?

CPU shouldn't be overloaded

Relevant logs and/or screenshots

Unknown

Output of checks

All good but it can't find the Go version?

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   2.3.6p384
Gem Version:    2.6.13
Bundler Version:1.13.7
Rake Version:   12.3.0
Redis Version:  3.2.11
Git Version:    2.14.3
Sidekiq Version:5.0.5
Go Version:     unknown
GitLab information
Version:        10.4.0
Revision:       128ef10
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            https://git.company.com
HTTP Clone URL: https://git.company.com/some-group/some-project.git
SSH Clone URL:  git@git-ssh.company.com:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: saml
GitLab Shell
Version:        5.11.0
Repository storage paths:

• default: /app/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

passing