Random 503 on creating merge requests (Gitlab::Git::CommandError (4:Deadline Exceeded))

Summary

Gitlab, upon creating a merge request, randomly generates 503s because of Gitlab::Git::CommandError (4:Deadline Exceeded).

Steps to reproduce

Try to create a merge request.

What is the current bug behavior?

Randomly, a 503 error appears after a timeout (50s).

What is the expected correct behavior?

Always create the merge request; when no error shows up, the merge request is created in a matter of seconds.

Relevant logs and/or screenshots

Gitlab::Git::CommandError (4:Deadline Exceeded):
  lib/gitlab/git/wraps_gitaly_errors.rb:13:in `rescue in wrapped_gitaly_errors'
  lib/gitlab/git/wraps_gitaly_errors.rb:6:in `wrapped_gitaly_errors'
  lib/gitlab/metrics/instrumentation.rb:161:in `block in wrapped_gitaly_errors'
  lib/gitlab/metrics/method_call.rb:36:in `measure'
  lib/gitlab/metrics/instrumentation.rb:161:in `wrapped_gitaly_errors'
  lib/gitlab/git/repository.rb:714:in `fetch_source_branch!'
  lib/gitlab/git/repository.rb:722:in `compare_source_branch'
  app/models/repository.rb:958:in `compare_source_branch'
  lib/gitlab/metrics/instrumentation.rb:161:in `block in compare_source_branch'
  lib/gitlab/metrics/method_call.rb:36:in `measure'
  lib/gitlab/metrics/instrumentation.rb:161:in `compare_source_branch'
  app/services/compare_service.rb:16:in `execute'
  app/services/merge_requests/build_service.rb:91:in `compare_branches'
  lib/gitlab/metrics/instrumentation.rb:161:in `block in compare_branches'
  lib/gitlab/metrics/method_call.rb:36:in `measure'
  lib/gitlab/metrics/instrumentation.rb:161:in `compare_branches'
  app/services/merge_requests/build_service.rb:26:in `execute'
  lib/gitlab/metrics/instrumentation.rb:161:in `block in execute'
  lib/gitlab/metrics/method_call.rb:36:in `measure'
  lib/gitlab/metrics/instrumentation.rb:161:in `execute'
  app/controllers/projects/merge_requests/creations_controller.rb:92:in `build_merge_request'
  lib/gitlab/i18n.rb:55:in `with_locale'
  lib/gitlab/i18n.rb:61:in `with_user_locale'
  app/controllers/application_controller.rb:423:in `set_locale'
  lib/gitlab/middleware/rails_queue_duration.rb:24:in `call'
  lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
  lib/gitlab/metrics/transaction.rb:55:in `run'
  lib/gitlab/metrics/rack_middleware.rb:17:in `call'
  lib/gitlab/middleware/multipart.rb:103:in `call'
  lib/gitlab/request_profiler/middleware.rb:16:in `call'
  ee/lib/gitlab/jira/middleware.rb:17:in `call'
  lib/gitlab/middleware/go.rb:20:in `call'
  lib/gitlab/etag_caching/middleware.rb:13:in `call'
  lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
  lib/gitlab/correlation_id.rb:15:in `use_id'
  lib/gitlab/middleware/correlation_id.rb:15:in `call'
  lib/gitlab/middleware/read_only/controller.rb:42:in `call'
  lib/gitlab/middleware/read_only.rb:18:in `call'
  lib/gitlab/middleware/basic_health_check.rb:25:in `call'
  lib/gitlab/request_context.rb:20:in `call'
  lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
  lib/gitlab/middleware/release_env.rb:13:in `call'

Results of GitLab environment info

System information
System:         Debian 9.8
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.5.3p105
Gem Version:    2.7.6
Bundler Version:1.16.6
Rake Version:   12.3.2
Redis Version:  3.2.12
Git Version:    2.18.1
Sidekiq Version:5.2.5
Go Version:     unknown

GitLab information
Version:        11.8.0-ee
Revision:       002a282
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
DB Version:     9.6.11
URL:            https://******.com
HTTP Clone URL: https://******.com/some-group/some-project.git
SSH Clone URL:  git@******.com:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: bitbucket

GitLab Shell
Version:        8.4.4
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
Hooks:          /opt/gitlab/embedded/service/gitlab-shell/hooks
Git:            /opt/gitlab/embedded/bin/git

What I already tried

Backuping the project before deleting and restoring it from backup; running housekeeping, both before and after restoring it. This did not change the problem; it can be circumvented, though, by retrying to create the merge request. Currently, retrying does not bring up the error again.

Edited by Zeger-Jan van de Weg