backup reliably fails with gitaly "connection failed" error
Summary
https://gitlab.zendesk.com/agent/tickets/92213
After upgrading from 10.2-ee to 10.5-ee, customer's regular backups reliably fail with the backtrace reported below. It does not fail on the same repo each time. Asking for detail from gitaly logs on the system.
Relevant logs and/or screenshots
GRPC::Unavailable: 14:Connect Failed
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grpc-1.8.3-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:31:in `check_status'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grpc-1.8.3-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:180:in `attach_status_results_and_complete_call'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grpc-1.8.3-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:372:in `request_response'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grpc-1.8.3-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:178:in `block in request_response'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grpc-1.8.3-x86_64-linux/src/ruby/lib/grpc/generic/interceptors.rb:170:in `intercept!'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grpc-1.8.3-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:177:in `request_response'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/grpc-1.8.3-x86_64-linux/src/ruby/lib/grpc/generic/service.rb:170:in `block (3 levels) in rpc_stub_class'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:127:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:17:in `exists?'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:142:in `block in exists?'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:232:in `block (2 levels) in migrate'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:270:in `allow_n_plus_1_calls'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:227:in `block in migrate'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/influx_db.rb:98:in `measure'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:225:in `migrate'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:140:in `exists?'
/opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:515:in `exists?'
/opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:78:in `block (2 levels) in cache_method'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:299:in `block in fetch'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:585:in `block in save_block_result_to_cache'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:547:in `block in instrument'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/notifications.rb:166:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:547:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:584:in `save_block_result_to_cache'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:299:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/lib/repository_cache.rb:20:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:1024:in `cache_method_output'
/opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:77:in `block in cache_method'
/opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:520:in `empty?'
/opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:78:in `block (2 levels) in cache_method'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:299:in `block in fetch'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:585:in `block in save_block_result_to_cache'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:547:in `block in instrument'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/notifications.rb:166:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:547:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:584:in `save_block_result_to_cache'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:299:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/lib/repository_cache.rb:20:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:1024:in `cache_method_output'
/opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:77:in `block in cache_method'
/opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:367:in `expire_emptiness_caches'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:198:in `empty_repo?'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:51:in `block in dump'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `block (2 levels) in find_each'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `block in find_each'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:124:in `find_in_batches'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:50:in `find_each'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/querying.rb:9:in `find_each'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:9:in `dump'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:82:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:251:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:251:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:251:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:195:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:188:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:181:in `invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:12:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:251:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:251:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:251:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:195:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:188:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:181:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:332:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor.rb:359:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:20:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/base.rb:440:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:11:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/exe/bundle:34:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/friendly_errors.rb:100:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/exe/bundle:26:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:repo:create
[DONE]