Random 503 on creating merge requests (Gitlab::Git::CommandError (4:Deadline Exceeded))
Summary
Gitlab, upon creating a merge request, randomly generates 503s because of Gitlab::Git::CommandError (4:Deadline Exceeded).
Steps to reproduce
Try to create a merge request.
What is the current bug behavior?
Randomly, a 503 error appears after a timeout (50s).
What is the expected correct behavior?
Always create the merge request; when no error shows up, the merge request is created in a matter of seconds.
Relevant logs and/or screenshots
Gitlab::Git::CommandError (4:Deadline Exceeded):
lib/gitlab/git/wraps_gitaly_errors.rb:13:in `rescue in wrapped_gitaly_errors'
lib/gitlab/git/wraps_gitaly_errors.rb:6:in `wrapped_gitaly_errors'
lib/gitlab/metrics/instrumentation.rb:161:in `block in wrapped_gitaly_errors'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `wrapped_gitaly_errors'
lib/gitlab/git/repository.rb:714:in `fetch_source_branch!'
lib/gitlab/git/repository.rb:722:in `compare_source_branch'
app/models/repository.rb:958:in `compare_source_branch'
lib/gitlab/metrics/instrumentation.rb:161:in `block in compare_source_branch'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `compare_source_branch'
app/services/compare_service.rb:16:in `execute'
app/services/merge_requests/build_service.rb:91:in `compare_branches'
lib/gitlab/metrics/instrumentation.rb:161:in `block in compare_branches'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `compare_branches'
app/services/merge_requests/build_service.rb:26:in `execute'
lib/gitlab/metrics/instrumentation.rb:161:in `block in execute'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `execute'
app/controllers/projects/merge_requests/creations_controller.rb:92:in `build_merge_request'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:423:in `set_locale'
lib/gitlab/middleware/rails_queue_duration.rb:24:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:55:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
ee/lib/gitlab/jira/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/correlation_id.rb:15:in `use_id'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:42:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
Results of GitLab environment info
System information
System: Debian 9.8
Proxy: no
Current User: git
Using RVM: no
Ruby Version: 2.5.3p105
Gem Version: 2.7.6
Bundler Version:1.16.6
Rake Version: 12.3.2
Redis Version: 3.2.12
Git Version: 2.18.1
Sidekiq Version:5.2.5
Go Version: unknown
GitLab information
Version: 11.8.0-ee
Revision: 002a282
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
DB Version: 9.6.11
URL: https://******.com
HTTP Clone URL: https://******.com/some-group/some-project.git
SSH Clone URL: git@******.com:some-group/some-project.git
Elasticsearch: no
Geo: no
Using LDAP: no
Using Omniauth: yes
Omniauth Providers: bitbucket
GitLab Shell
Version: 8.4.4
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks
Git: /opt/gitlab/embedded/bin/git
What I already tried
Backuping the project before deleting and restoring it from backup; running housekeeping, both before and after restoring it. This did not change the problem; it can be circumvented, though, by retrying to create the merge request. Currently, retrying does not bring up the error again.
Edited by Zeger-Jan van de Weg