GitLab performance degradation

Summary

High CPU load. Web UI is so slow. Catch errors while cloning repos

Steps to reproduce

Run gitlab instance with 1000 users

  1. HW: HP DL 360p Gen8 with 256GB ram + 4 SSD (Samsung PRO series) in raid 10 (hardware raid controller HP)

  2. OS: Ubuntu 16.04 with 4.15.0-29 kernel

  3. Gitlab running in docker container

docker version
Client: Docker Engine - Community
 Version:           19.03.1
 API version:       1.39 (downgraded from 1.40)
 Go version:        go1.12.5
 Git commit:        74b1e89e8a
 Built:             Thu Jul 25 21:21:35 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.4
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       d14af54
  Built:            Wed Mar 27 18:01:48 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Inside the container:

root@gitlab:/# gitlab-rake gitlab:env:info

System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   2.5.3p105
Gem Version:    2.7.9
Bundler Version:1.17.3
Rake Version:   12.3.2
Redis Version:  3.2.12
Git Version:    2.21.0
Sidekiq Version:5.2.7
Go Version:     unknown

GitLab information
Version:        11.11.7
Revision:       f1399b947ef
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     9.6.11
URL:            https://gitlab*****.com
HTTP Clone URL: https://gitlab*****.com/some-group/some-project.git
SSH Clone URL:  git@gitlab*****.com:some-group/some-project.git
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        9.1.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git



root@gitlab:/#  gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 9.1.0 ? ... OK (9.1.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK

Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
        DN: cn='****',ou=****,ou=***,ou=***,ou=*****,ou=*****,dc=***,dc=***,dc=***,dc=com     sAMAccountName: '********'$
        . ***
		. ***
		. ***
        DN: cn=****,ou=****,ou=***,ou=***,ou=***,dc=***,dc=***,dc=***,dc=com  sAMAccountName:  **********

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
22/2 ... yes
50/4 ... yes
35/5 ... yes
50/6 ... yes
35/8 ... yes
*
*
*
142/1095 ... yes
1424/1096 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.5.3)
Git version >= 2.21.0 ? ... yes (2.21.0)
Git user has default SSH configuration? ... yes
Active users: ... 1318

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished
  1. Gitlab config
external_url 'https://gitlab*******.com'
nginx['redirect_http_to_https'] = true
nginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab.key"

gitlab_rails['ldap_enabled'] = true
gitlab_rails['ldap_host'] = '********'
gitlab_rails['ldap_port'] = ******
gitlab_rails['ldap_uid'] = 'sAMAccountName'
gitlab_rails['ldap_method'] = 'ssl' # 'ssl' or 'plain'
gitlab_rails['ldap_bind_dn'] = '*******'
gitlab_rails['ldap_password'] = '*******'
gitlab_rails['ldap_allow_username_or_email_login'] = true
gitlab_rails['ldap_base'] = '********'
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "********"
gitlab_rails['smtp_port'] = 25
#gitlab_rails['smtp_user_name'] = "*******"
#gitlab_rails['smtp_password'] = "smtp password"
gitlab_rails['smtp_domain'] = "*******"
gitlab_rails['smtp_authentication'] = false
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['gitlab_email_from'] = '******'
gitlab_rails['gitlab_email_reply_to'] = '******
gitlab_rails['lfs_enabled'] = true
prometheus_monitoring['enable'] = false
gitlab_monitor['enable'] = false
gitaly['ruby_num_workers'] = 3
unicorn['worker_processes'] = 43
unicorn['worker_timeout'] = 50

  1. htop output gitlab_load

  2. perf report for gitaly process gitlab_gitaly_perf

  3. From time to time we catch the next problem, while cloning repos:

gitaly-upload-pack: fatal: error: %v
fatal: Could not read from remote repository.
  1. Gitlab UI is so slow(((
Edited by Alexander Tsyplenkov