High CPU usage following OpenSSH upgrade (via OS upgrade) even during off-hours
Summary
GitLab 13.5 was running fine then when upgraded to 13.12 started seeing high CPU usage including during off hours with minimal traffic. Tried upgrading to 14.0 to see if that would help but no luck.
GitLab upgrade just happened to coincide with when sshd issue manifested itself.
Steps to reproduce
- Oracle Linux 7.9
Upgrade to 13.12.15 then 14.0.12- Run Gitaly on separate server
What is the current bug behavior?
High CPU usage even under minimal load. Was working fine with 13.5. worked fine with older OpenSSH.
What is the expected correct behavior?
GitLab perform as well or better in newer versions as it does with older versions.
Relevant logs and/or screenshots
htop
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: OracleServer 7.9 Current User: git Using RVM: no Ruby Version: 2.7.2p137 Gem Version: 3.1.4 Bundler Version:2.1.4 Rake Version: 13.0.3 Redis Version: 6.0.14 Git Version: 2.32.0 Sidekiq Version:5.2.9 Go Version: unknown GitLab information Version: 14.0.12 Revision: def69dde9f1 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 12.7 URL: x HTTP Clone URL: x SSH Clone URL: x Using LDAP: no Using Omniauth: yes Omniauth Providers: saml GitLab Shell Version: 13.19.1 Repository storage paths: - prod-gitalya-0: /var/opt/gitlab/git-data/repositories - prod-gitalyb-0: /var/opt/gitlab/git-data/repositories - default: /var/opt/gitlab/git-data/repositories - nfs: /var/opt/gitlab/git-data/repositories - nfs2-prod: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git NOTE: above used to use NFS but no longer do and just have old names
Results of GitLab application Check
Expand for output related to the GitLab application check
$ sudo gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.19.1 ? ... OK (13.19.1) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... prod-gitalya-0 ... OK prod-gitalyb-0 ... OK default ... OK nfs ... OK nfs2-prod ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/2
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Checking Reply by email ...
IMAP server credentials are correct? ... Checking x@x.com yes Init.d configured correctly? ... skipped MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... All green Redis version >= 5.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.2) Git version >= 2.31.0 ? ... yes (2.32.0) Git user has default SSH configuration? ... yes Active users: ... xxx Is authorized keys file accessible? ... skipped (authorized keys not enabled) GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished