"slice bounds out of range" in gitaly during restore
Summary
I am following the https://docs.gitlab.com/ce/raketasks/backup_restore.html page to move a gitlab installation (omnibus) from one server to another.
Both servers have the same version of gitlab installed.
The backup process completes just fine, I copy the tar to the new server and execute the restore.
It restores the database and a quite large amount of repositories, but the procedure hangs when importing a specific repository. I tried to make the backup again and to restore it multiple times, with the same behaviour.
Steps to reproduce
in the old server
gitlab-rake gitlab:backup:create SKIP=registry DIRECTORY=/tmp
scp /tmp/<filename> root@<newserver>:/var/opt/gitlab/backups
in the new server
chown git:git /var/opt/gitlab/backups/*
gitlab-rake gitlab:backup:restore BACKUP=<filename> force=yes --trace
What is the current bug behavior?
The process exits with the following error:
* internals/projectname1 ... [DONE]
* internals/projectname2 ... [DONE]
* internals/projectname3 ... Error: 14:Socket closed
[Failed] restoring internals/projectname3 repository
* clients/projectname4 ... rake aborted!
GRPC::Unavailable: 14:Connect Failed
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.19.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:31:in `check_status'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.19.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:181:in `attach_status_results_and_complete_call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.19.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:377:in `request_response'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.19.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:178:in `block in request_response'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.19.0-x86_64-linux/src/ruby/lib/grpc/generic/interceptors.rb:170:in `intercept!'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.19.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:177:in `request_response'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grpc-1.19.0-x86_64-linux/src/ruby/lib/grpc/generic/service.rb:170:in `block (3 levels) in rpc_stub_class'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:162:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/namespace_service.rb:37:in `gitaly_client_call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/namespace_service.rb:19:in `add'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/shell.rb:266:in `block in add_namespace'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:305:in `allow_n_plus_1_calls'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/shell.rb:265:in `add_namespace'
/opt/gitlab/embedded/service/gitlab-rails/app/models/storage/legacy_project.rb:29:in `ensure_storage_path_exists'
/opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:67:in `ensure_storage_path_exists'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:83:in `block in restore'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:63:in `block (2 levels) in find_each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:63:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:63:in `block in find_each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:129:in `block in find_in_batches'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:230:in `block in in_batches'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:214:in `loop'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:214:in `in_batches'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:128:in `find_in_batches'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:62:in `find_each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.1.7/lib/active_record/querying.rb:9:in `find_each'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:80:in `restore'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:87:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:194:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:183:in `invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:60:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:194:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:183:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:repo:restore
What is the expected correct behavior?
I expected the rake task to successfully import the backup in the new server.
Relevant logs and/or screenshots
I found this in gitaly's "current" log file and I think it's relevant:
2019-05-28_15:13:38.68992 time="2019-05-28T15:13:38Z" level=error msg="Cloning into bare repository '/srv/gitlab/repositories/internals/projectname3.git'...\\n" grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-web grpc.method=CreateRepositoryFromBundle grpc.request.fullMethod=/gitaly.RepositoryService/CreateRepositoryFromBundle grpc.request.glProjectPath=internals/projectname3 grpc.request.glRepository=project-298 grpc.request.repoPath=internals/projectname3.git grpc.request.repoStorage=default grpc.request.topLevelGroup=internals grpc.service=gitaly.RepositoryService grpc.start_time="2019-05-28T15:13:36Z" peer.address=@ span.kind=server system=grpc
2019-05-28_15:13:38.83701 panic: runtime error: slice bounds out of range
2019-05-28_15:13:38.83707
2019-05-28_15:13:38.83708 goroutine 4712 [running]:
2019-05-28_15:13:38.83785 gitlab.com/gitlab-org/gitaly/internal/command.writeLines(0xddc760, 0xc0000a0468, 0xdd8bc0, 0xc0000a04f0, 0xc000199440, 0x2710)
2019-05-28_15:13:38.83802 /var/cache/omnibus/src/gitaly/_build/src/gitlab.com/gitlab-org/gitaly/internal/command/command.go:313 +0x54c
2019-05-28_15:13:38.83829 created by gitlab.com/gitlab-org/gitaly/internal/command.escapeNewlineWriter
2019-05-28_15:13:38.83830 /var/cache/omnibus/src/gitaly/_build/src/gitlab.com/gitlab-org/gitaly/internal/command/command.go:278 +0x18c
2019-05-28_15:13:38.84133 time="2019-05-28T15:13:38Z" level=warning msg="forwarding signal" gitaly=15652 signal="child exited" wrapper=15645
2019-05-28_15:13:38.95313 time="2019-05-28T15:13:38Z" level=error msg="wrapper for gitaly shutting down" gitaly=15652 wrapper=15645
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Old server's GitLab environment info
System information System: Ubuntu 18.04 Current User: git Using RVM: no Ruby Version: 2.5.3p105 Gem Version: 2.7.9 Bundler Version:1.17.3 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.21.0 Sidekiq Version:5.2.7 Go Version: unknownGitLab information Version: 11.11.0 Revision: 3e8ca2fb781 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.8 URL: https://git.mvlabs.it HTTP Clone URL: https://git.mvlabs.it/some-group/some-project.git SSH Clone URL: ssh://git@git.mvlabs.it:4222/some-group/some-project.git Using LDAP: yes Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 9.1.0 Repository storage paths:
- default: /srv/gitlab/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
New server's GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.5.3p105 Gem Version: 2.7.9 Bundler Version:1.17.3 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.21.0 Sidekiq Version:5.2.7 Go Version: unknownGitLab information Version: 11.11.0 Revision: 3e8ca2fb781 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 9.6.13 URL: https://git.mvlabs.it HTTP Clone URL: https://git.mvlabs.it/some-group/some-project.git SSH Clone URL: ssh://git@git.mvlabs.it:4222/some-group/some-project.git Using LDAP: yes Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 9.1.0 Repository storage paths:
- default: /srv/gitlab/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)