"slice bounds out of range" in gitaly during restore

Summary

I am following the https://docs.gitlab.com/ce/raketasks/backup_restore.html page to move a gitlab installation (omnibus) from one server to another.

Both servers have the same version of gitlab installed.

The backup process completes just fine, I copy the tar to the new server and execute the restore.

It restores the database and a quite large amount of repositories, but the procedure hangs when importing a specific repository. I tried to make the backup again and to restore it multiple times, with the same behaviour.

Steps to reproduce

in the old server

gitlab-rake gitlab:backup:create SKIP=registry DIRECTORY=/tmp
scp /tmp/<filename> root@<newserver>:/var/opt/gitlab/backups

in the new server

chown git:git /var/opt/gitlab/backups/*
gitlab-rake gitlab:backup:restore BACKUP=<filename> force=yes --trace

What is the current bug behavior?

The process exits with the following error:

 * internals/projectname1 ... [DONE]
 * internals/projectname2 ... [DONE]
 * internals/projectname3 ... Error: 14:Socket closed
[Failed] restoring internals/projectname3 repository
 * clients/projectname4 ... rake aborted!
GRPC::Unavailable: 14:Connect Failed
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.19.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:31:in `check_status'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.19.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:181:in `attach_status_results_and_complete_call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.19.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:377:in `request_response'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.19.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:178:in `block in request_response'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.19.0-x86_64-linux/src/ruby/lib/grpc/generic/interceptors.rb:170:in `intercept!'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.19.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:177:in `request_response'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.19.0-x86_64-linux/src/ruby/lib/grpc/generic/service.rb:170:in `block (3 levels) in rpc_stub_class'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:162:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/namespace_service.rb:37:in `gitaly_client_call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/namespace_service.rb:19:in `add'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/shell.rb:266:in `block in add_namespace'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:305:in `allow_n_plus_1_calls'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/shell.rb:265:in `add_namespace'
/opt/gitlab/embedded/service/gitlab-rails/app/models/storage/legacy_project.rb:29:in `ensure_storage_path_exists'
/opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:67:in `ensure_storage_path_exists'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:83:in `block in restore'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:63:in `block (2 levels) in find_each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:63:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:63:in `block in find_each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:129:in `block in find_in_batches'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:230:in `block in in_batches'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:214:in `loop'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:214:in `in_batches'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:128:in `find_in_batches'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:62:in `find_each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.1.7/lib/active_record/querying.rb:9:in `find_each'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:80:in `restore'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:87:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:194:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:183:in `invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:60:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:194:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:183:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:repo:restore

What is the expected correct behavior?

I expected the rake task to successfully import the backup in the new server.

Relevant logs and/or screenshots

I found this in gitaly's "current" log file and I think it's relevant:

2019-05-28_15:13:38.68992 time="2019-05-28T15:13:38Z" level=error msg="Cloning into bare repository '/srv/gitlab/repositories/internals/projectname3.git'...\\n" grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-web grpc.method=CreateRepositoryFromBundle grpc.request.fullMethod=/gitaly.RepositoryService/CreateRepositoryFromBundle grpc.request.glProjectPath=internals/projectname3 grpc.request.glRepository=project-298 grpc.request.repoPath=internals/projectname3.git grpc.request.repoStorage=default grpc.request.topLevelGroup=internals grpc.service=gitaly.RepositoryService grpc.start_time="2019-05-28T15:13:36Z" peer.address=@ span.kind=server system=grpc
2019-05-28_15:13:38.83701 panic: runtime error: slice bounds out of range
2019-05-28_15:13:38.83707
2019-05-28_15:13:38.83708 goroutine 4712 [running]:
2019-05-28_15:13:38.83785 gitlab.com/gitlab-org/gitaly/internal/command.writeLines(0xddc760, 0xc0000a0468, 0xdd8bc0, 0xc0000a04f0, 0xc000199440, 0x2710)
2019-05-28_15:13:38.83802       /var/cache/omnibus/src/gitaly/_build/src/gitlab.com/gitlab-org/gitaly/internal/command/command.go:313 +0x54c
2019-05-28_15:13:38.83829 created by gitlab.com/gitlab-org/gitaly/internal/command.escapeNewlineWriter
2019-05-28_15:13:38.83830       /var/cache/omnibus/src/gitaly/_build/src/gitlab.com/gitlab-org/gitaly/internal/command/command.go:278 +0x18c
2019-05-28_15:13:38.84133 time="2019-05-28T15:13:38Z" level=warning msg="forwarding signal" gitaly=15652 signal="child exited" wrapper=15645
2019-05-28_15:13:38.95313 time="2019-05-28T15:13:38Z" level=error msg="wrapper for gitaly shutting down" gitaly=15652 wrapper=15645

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Old server's GitLab environment info

System information
System:         Ubuntu 18.04
Current User:   git
Using RVM:      no
Ruby Version:   2.5.3p105
Gem Version:    2.7.9
Bundler Version:1.17.3
Rake Version:   12.3.2
Redis Version:  3.2.12
Git Version:    2.21.0
Sidekiq Version:5.2.7
Go Version:     unknown
GitLab information
Version:        11.11.0
Revision:       3e8ca2fb781
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     10.8
URL:            https://git.mvlabs.it
HTTP Clone URL: https://git.mvlabs.it/some-group/some-project.git
SSH Clone URL:  ssh://git@git.mvlabs.it:4222/some-group/some-project.git
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version:        9.1.0
Repository storage paths:

• default: /srv/gitlab/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

New server's GitLab environment info

System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   2.5.3p105
Gem Version:    2.7.9
Bundler Version:1.17.3
Rake Version:   12.3.2
Redis Version:  3.2.12
Git Version:    2.21.0
Sidekiq Version:5.2.7
Go Version:     unknown
GitLab information
Version:        11.11.0
Revision:       3e8ca2fb781
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     9.6.13
URL:            https://git.mvlabs.it
HTTP Clone URL: https://git.mvlabs.it/some-group/some-project.git
SSH Clone URL:  ssh://git@git.mvlabs.it:4222/some-group/some-project.git
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version:        9.1.0
Repository storage paths:

• default: /srv/gitlab/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)