Listing project contributors times out (after 60 seconds) with "Error fetching contributors data" error.
Summary
GitLab 11.2.3-ee (gitlab-ce@aadca994104734b14ca18e6c10d5e3c54a435b4a)
GitLab Shell 8.1.1
GitLab Workhorse v5.1.0
GitLab API v4
Ruby 2.4.4p296
Rails 4.2.10
Gitaly 0.117.2
GitLab EE Omnibus in AWS cloud + external postgres and redis.
Currently we have four t3.large instances running GitLab, Registry and Gitaly.
We have a number of projects we imported from GitHub.
Impact:
some of the imported from GitHub projects. We assume that impacted projects have relatively big history.
Not all of the projects experience this issue.
When we try to open project contributors page it times out after ~60 seconds with "Error fetching contributors data" error message. And gets stuck on "Building repository graph. Please wait a moment, this page will automatically refresh when ready."
What have been done
- Increased Gitaly timeout settings in "Admin Area" to 3 minutes.
- Increased
unicorn['worker_timeout'] = 120
- Increased
gitaly['ruby_num_workers'] = 4
- Increased
gitlab_workhorse['proxy_headers_timeout'] = "2m0s"
CPU and memory load/usage is low.
Nothing obvious in Gitlab logs, but this:
2018-09-19_17:54:07.06246 gitlab-server-01 gitlab-workhorse: time="2018-09-19T17:54:07Z" level=error msg=error correlation-id=E1kgV0oC9oa error="badgateway: failed after 59s: context canceled" method=GET uri="/project_name/graphs/master?format=json"
2018-09-19_17:54:07.06267 gitlab-server-01 gitlab-workhorse: gitlab.example.com @ - - [2018/09/19:17:54:07 +0000] "GET /project_name/graphs/master?format=json HTTP/1.1" 502 2940 "https://gitlab.example.com/project_name/graphs/master" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36" 59.377
Thank you in advance!
Steps to reproduce
Project -> Repository -> Contributors (click)
Configuration used (only relevant to the reported issue)
external_url 'gitlab.example.com'
registry_external_url 'registry.example.com'
nginx['listen_port'] = 80
nginx['listen_https'] = false
nginx['proxy_set_headers'] = {
"X-Forwarded-Proto" => "https",
"X-Forwarded-Ssl" => "on"
}
nginx['real_ip_trusted_addresses'] = [ '${ELB_IPs}' ]
nginx['real_ip_header'] = 'X-Forwarded-For'
nginx['real_ip_recursive'] = 'on'
registry_nginx['listen_port'] = 80
registry_nginx['listen_https'] = false
registry_nginx['proxy_set_headers'] = {
'X-Forwarded-Proto' => 'https',
'X-Forwarded-Ssl' => 'on'
}
registry_nginx['real_ip_trusted_addresses'] = [ '${ELB_IPs}' ]
registry_nginx['real_ip_header'] = 'X-Forwarded-For'
registry_nginx['real_ip_recursive'] = 'on'
high_availability['mountpoint'] = ['${MOUNT_POINTS}']
postgresql['enable'] = false
redis['enable'] = false
gitlab_rails['db_adapter'] = "postgresql"
gitlab_rails['auto_migrate'] = false
gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '${ELB_IPs}']
gitlab_rails['smtp_enable'] = true
gitlab_rails['gitlab_default_can_create_group'] = false
gitlab_rails['rake_cache_clear'] = false
unicorn['worker_timeout'] = 120
unicorn['worker_processes'] = 6
gitlab_workhorse['proxy_headers_timeout'] = "2m0s"
Current behavior
For some projects, Contributors page times out after ~60 seconds with "Error fetching contributors data" error message.
Expected behavior
normal operation, show graphs and contributors information.
Versions
- Platform:
- Cloud: AWS
Relevant logs
2018-09-19_17:54:07.06246 gitlab-server-01 gitlab-workhorse: time="2018-09-19T17:54:07Z" level=error msg=error correlation-id=E1kgV0oC9oa error="badgateway: failed after 59s: context canceled" method=GET uri="/project_name/graphs/master?format=json"
2018-09-19_17:54:07.06267 gitlab-server-01 gitlab-workhorse: gitlab.example.com @ - - [2018/09/19:17:54:07 +0000] "GET /project_name/graphs/master?format=json HTTP/1.1" 502 2940 "https://gitlab.example.com/project_name/graphs/master" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36" 59.377