### Summary Hello Everyone, I have gitlab with in docker container. I am following the backup and restore process from https://docs.gitlab.com/ee/raketasks/backup_restore.html documentation. When I start doing the restore, Initially it starts unpacking, restoring repositories and suddenly it fails with the GRPC error. ### Steps to reproduce I have follwed steps provided for backup and restore from https://docs.gitlab.com/ee/raketasks/backup_restore.html 1. Created backup from gitlab with in docker container 2. Copied the tar file from one server to another server 3. Copied tar file to /var/opt/gitlab/backups path which is again gitlab with in docker container. 4. Backup tar file is owned by git 5. gitlab-ctl stop unicorn 6. gitlab-ctl stop sidekiq 7. gitlab-ctl status 8. gitlab-rake gitlab:backup:restore BACKUP=backup_file ### What is the current *bug* behavior? (What actually happens) ### What is the expected *correct* behavior? (What you should see instead) ### Relevant logs Restore with --trace option ``` * HQITD51FSWEBD/H5-EMVFunding ... [DONE] * Starfire/Starfire ... [DONE] * Harvest/SourceSafeHQDATA03 ... Error: 14:OS Error [Failed] restoring Harvest/SourceSafeHQDATA03 repository * HQITD51FSWEBD/H5-CAPS ... rake aborted! GRPC::Unavailable: 14:OS Error /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:31:in `check_status' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:180:in `attach_status_results_and_complete_call' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:372:in `request_response' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:178:in `block in request_response' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/interceptors.rb:170:in `intercept!' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:177:in `request_response' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/service.rb:170:in `block (3 levels) in rpc_stub_class' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:134:in `call' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/namespace_service.rb:35:in `gitaly_client_call' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/namespace_service.rb:17:in `add' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/shell.rb:280:in `add_namespace' /opt/gitlab/embedded/service/gitlab-rails/app/models/storage/legacy_project.rb:29:in `ensure_storage_path_exists' /opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:60:in `ensure_storage_path_exists' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:81:in `block in restore' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `block (2 levels) in find_each' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `each' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `block in find_each' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:124:in `find_in_batches' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:50:in `find_each' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/querying.rb:9:in `find_each' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:78:in `restore' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:87:in `block (4 levels) in <top (required)>' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `block in execute' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `each' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `execute' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:213:in `block in invoke_with_call_chain' /opt/gitlab/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:182:in `invoke' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:60:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `block in execute' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `each' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `execute' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:213:in `block in invoke_with_call_chain' /opt/gitlab/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:182:in `invoke' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:160:in `invoke_task' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `block (2 levels) in top_level' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `each' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `block in top_level' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:125:in `run_with_threads' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:110:in `top_level' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:83:in `block in run' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:186:in `standard_exception_handling' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:80:in `run' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/exe/rake:27:in `<top (required)>' /opt/gitlab/embedded/bin/rake:23:in `load' /opt/gitlab/embedded/bin/rake:23:in `<top (required)>' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:74:in `load' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:74:in `kernel_load' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:28:in `run' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:424:in `exec' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:27:in `dispatch' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:18:in `start' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/exe/bundle:30:in `block in <top (required)>' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/exe/bundle:22:in `<top (required)>' /opt/gitlab/embedded/bin/bundle:23:in `load' /opt/gitlab/embedded/bin/bundle:23:in `<main>' Tasks: TOP => gitlab:backup:repo:restore ``` I am pasting** gitaly log** output before it failed. In the log file as well..it says it cloned bare repositories into git directory. Please look into the below ``` 2019-01-14_19:32:29.60104 time="2019-01-14T19:32:29Z" level=info msg="finished unary call" grpc.code=OK grpc.meta.auth_version=v1 grpc.meta.client_name=gitlab-web grpc.method=AddNamespace grpc.request.Name=CreditTeam grpc.request.StorageName=default grpc.service=gitaly.NamespaceService grpc.time_ms=0 peer.address=@ span.kind=server system=grpc 2019-01-14_19:32:31.44505 time="2019-01-14T19:32:31Z" level=info msg="Cloning into bare repository '/var/opt/gitlab/git-data/repositories/CreditTeam/mapps.git'..." grpc.meta.auth_version=v1 grpc.meta.client_name=gitlab-web grpc.method=CreateRepositoryFromBundle grpc.request.glRepository=project-95 grpc.request.repoPath=CreditTeam/mapps.git grpc.request.repoStorage=default grpc.request.topLevelGroup=CreditTeam grpc.service=gitaly.RepositoryService peer.address=@ span.kind=server system=grpc 2019-01-14_19:32:33.00589 time="2019-01-14T19:32:33Z" level=warning msg=exited error="signal: killed" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 933 /tmp/gitaly-ruby696213238/socket.0]" supervisor.name=gitaly-ruby.0 2019-01-14_19:32:33.00918 time="2019-01-14T19:32:33Z" level=warning msg=exited error="signal: killed" supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 933 /tmp/gitaly-ruby696213238/socket.1]" supervisor.name=gitaly-ruby.1 2019-01-14_19:32:33.01388 time="2019-01-14T19:32:33Z" level=fatal msg="shutting down" error="received signal \"terminated\"" ``` ### Details of package version <details> <summary>Provide the package version installation details</summary> <pre> (For Debian, Ubuntu based systems, paste the output of: `dpkg-query -l "gitlab-*"`) (For RHEL based systems (CentOS, RHEL, OL, Scientific, OpenSUSE, SLES), paste the output of: `rpm -qa | grep 'gitlab'`) (If you can't provide these details, see https://about.gitlab.com/getting-help/ page on where to ask your question.) </pre> </details> ### Environment details * Operating System: `SUSE Linux Enterprise Server 12 SP3` * Installation Target, remove incorrect values: * Other: `Gitlab with in docker container` * Installation Type, remove incorrect values: * `New omnibus installation using docker image` * Is there any other software running on the machine: `Standalone Postgres and Redis docker containers [ I am not using them with gitlab docker container]` * Is this a single or multiple node installation? `Single` * Resources * CPU: `cpu cores 1` * Memory total: `16GB` ### Configuration details gitlab_rails['time_zone'] = 'America/Chicago' gitlab_rails['gitlab_default_can_create_group'] = false gitlab_rails['gitlab_default_projects_features_snippets'] = false gitlab_rails['gitlab_default_projects_features_builds'] = false LDAP enabled Bundled Nginx is being used with ssl certs Everything else is default