Backup repository file with rake aborted after upgrade to 11.1
Summary
After upgrade to milestone %11.1, backup of repository is failing.
Steps to reproduce
Simply launch /usr/bin/gitlab-rake gitlab:backup:create
will report this error on the first repository that is trying to backup:
GRPC::Core::CallError: call#run_batch failed somehow
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:253:in `run_batch'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:253:in `remote_read'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:326:in `block in each_remote_read_then_finish'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:325:in `loop'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:325:in `each_remote_read_then_finish'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:348:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:348:in `block in gitaly_fetch_stream_to_file'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:347:in `open'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:347:in `gitaly_fetch_stream_to_file'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:201:in `create_bundle'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:67:in `backup_project_gitaly'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:53:in `block in backup_project'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:289:in `block (2 levels) in migrate'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:329:in `allow_n_plus_1_calls'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:282:in `block in migrate'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/influx_db.rb:98:in `measure'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:280:in `migrate'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:285:in `gitaly_migrate'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:51:in `backup_project'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:27:in `block in dump'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `block (2 levels) in find_each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `block in find_each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:124:in `find_in_batches'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:50:in `find_each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/querying.rb:9:in `find_each'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:17:in `dump'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:80:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:182:in `invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:11:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:182:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:424:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:repo:create
What is the current bug behavior?
Backup is failing and it does not produce a consistent backup.
What is the expected correct behavior?
Backup is working as usual.
/label ~bug
Edited by Gian Luca Dalla Torre