HTTP requests are slow on selfhosted GitLab-ee
Summary
Most of the pages take more than 2-3 seconds to load. Profiling bar shows very small times for databases, so it looks like the most time GitLab spends on rendering the page or just waiting for some timeouts. The problem is persistent, no matter how loaded is the server. I was investigating this while no one else was using the instance and there were no difference in performance in comparison with busy hours.
The server is big enough to handle our small GitLab instance, it has 4 CPUs and 16Gb memory.
API requests, like /api/v4/projects/104/issues/38
, take time in range 0.3-0.6 seconds, spiking sometimes to 1.5 seconds, but very rarely.
I also tried to send requests with x-profile-token
in header, but I wasn't able to get any insights from the resulting report.
Some pages are loaded under one second. Those are mostly pages with disabled functionality or no content, like empty package registry or empty kubernetes clusters page. But I also saw project members page was loaded fast (3 members in the project).
Steps to reproduce
Not sure if it's the case for any selfhosted GitLab instance, but just opening any page which has any content. The problem is persistent on issues pages.
What is the current bug behavior?
HTTP requests are a couple of seconds slower than they should be.
What is the expected correct behavior?
Pages are being loaded in time under 2 seconds.
Relevant logs and/or screenshots
Here are some examples.
Project issues page (there are 39 issues in total):
Requesting the same issue via API:
$ curl --header "PRIVATE-TOKEN: $GITLAB_PRIVATE_TOKEN" https://git.xxx.com/api/v4/projects/104/issues/38 -s -o /dev/null -w "@curl-format.txt"
time_namelookup: 0,028116s
time_connect: 0,051923s
time_appconnect: 0,103141s
time_pretransfer: 0,103398s
time_redirect: 0,000000s
time_starttransfer: 0,230732s
----------
time_total: 0,230876s
$ curl --header "PRIVATE-TOKEN: $GITLAB_PRIVATE_TOKEN" https://git.xxx.com/api/v4/projects/104/issues/38 -s -o /dev/null -w "@curl-format.txt"
time_namelookup: 0,028490s
time_connect: 0,051474s
time_appconnect: 0,103249s
time_pretransfer: 0,103503s
time_redirect: 0,000000s
time_starttransfer: 0,463101s
----------
time_total: 0,463158s
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Debian 10 Proxy: no Current User: git Using RVM: no Ruby Version: 2.6.6p146 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 5.0.9 Git Version: 2.28.0 Sidekiq Version:5.2.9 Go Version: unknown GitLab information Version: 13.4.4-ee Revision: 4196ccb4738 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 11.9 URL: https://git.xxx.com HTTP Clone URL: https://git.xxx.com/some-group/some-project.git SSH Clone URL: git@git.xxx.com:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers: google_oauth2 GitLab Shell Version: 13.7.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.7.0 ? ... OK (13.7.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 4/1 ... yes 4/2 ... yes 4/3 ... yes 4/4 ... yes 4/5 ... yes 4/6 ... yes 4/8 ... yes 4/9 ... yes 4/10 ... yes 4/11 ... yes 4/13 ... yes 4/14 ... yes 4/15 ... yes 15/16 ... yes 4/17 ... yes 4/18 ... yes 5/19 ... yes 5/20 ... yes 5/21 ... yes 5/22 ... yes 4/23 ... yes 25/24 ... yes 26/25 ... yes 25/27 ... yes 25/29 ... yes 26/30 ... yes 37/33 ... yes 27/35 ... yes 27/36 ... yes 25/37 ... yes 26/38 ... yes 4/40 ... yes 10/41 ... yes 5/43 ... yes 32/44 ... yes 27/45 ... yes 6/48 ... yes 5/49 ... yes 4/50 ... yes 5/52 ... yes 4/53 ... yes 37/54 ... yes 35/55 ... yes 36/56 ... yes 27/57 ... yes 4/58 ... yes 37/59 ... yes 4/60 ... yes 32/61 ... yes 5/62 ... yes 32/63 ... yes 27/64 ... yes 32/65 ... yes 32/67 ... yes 36/68 ... yes 27/69 ... yes 37/70 ... yes 5/72 ... yes 10/73 ... yes 37/74 ... yes 37/75 ... yes 37/76 ... yes 37/77 ... yes 37/78 ... yes 37/79 ... yes 26/80 ... yes 25/81 ... yes 37/82 ... yes 37/83 ... yes 35/84 ... yes 15/85 ... yes 15/86 ... yes 37/87 ... yes 15/88 ... yes 37/89 ... yes 21/90 ... yes 9/91 ... yes 25/93 ... yes 36/94 ... yes 37/97 ... yes 37/98 ... yes 4/101 ... yes 37/102 ... yes 4/103 ... yes 26/104 ... yes 26/105 ... yes 26/106 ... yes 36/107 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.6) Git version >= 2.24.0 ? ... yes (2.28.0) Git user has default SSH configuration? ... yes Active users: ... 12 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Elasticsearch version 6.x - 7.x? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished