Gitlab Backup fails with Gitlab::Git::CommandError: 7:permission denied

Category:Backup/Restore of GitLab instances backend ~bug

We have Gitlab installed with the Gitlab Helm Chart Version gitlab-3.1.4 and the app Version 12.8.6. If we configure the task runners backup as follows, the backup fails.

gitlab:
## doc/charts/globals.md#configure-postgresql-settings
  task-runner:
    backups:
      cron:
        enabled: true
        schedule: "0 1 * * *"
        extraArgs: "--skip db,artifacts,uploads,lfs,registry,packages"

The error Logs are as follows:

Begin parsing .erb files from /var/opt/gitlab/templates
Writing /srv/gitlab/config/resque.yml
Writing /srv/gitlab/config/gitlab.yml
Writing /srv/gitlab/config/database.yml
Copying other config files found in /var/opt/gitlab/templates
Copying smtp_settings.rb into /srv/gitlab/config
Attempting to run '/bin/bash -c cp /etc/gitlab/.s3cfg $HOME/.s3cfg && backup-utility --skip db,artifacts,uploads,lfs,registry,packages' as a main process
rake aborted!
Gitlab::Git::CommandError: 7:permission denied
/srv/gitlab/lib/gitlab/git/wraps_gitaly_errors.rb:13:in `rescue in wrapped_gitaly_errors'
/srv/gitlab/lib/gitlab/git/wraps_gitaly_errors.rb:6:in `wrapped_gitaly_errors'
/srv/gitlab/lib/gitlab/git/repository.rb:1021:in `uncached_has_local_branches?'
/srv/gitlab/lib/gitlab/git/repository.rb:161:in `block in has_local_branches?'
/srv/gitlab/lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
/srv/gitlab/lib/gitlab/git/repository.rb:160:in `has_local_branches?'
/srv/gitlab/app/models/repository.rb:560:in `has_visible_content?'
/srv/gitlab/lib/gitlab/repository_cache_adapter.rb:84:in `block (2 levels) in cache_method_asymmetrically'
/srv/gitlab/lib/gitlab/repository_cache.rb:44:in `fetch_without_caching_false'
/srv/gitlab/lib/gitlab/repository_cache_adapter.rb:179:in `block (2 levels) in cache_method_output_asymmetrically'
/srv/gitlab/lib/gitlab/null_request_store.rb:34:in `fetch'
/srv/gitlab/lib/gitlab/safe_request_store.rb:12:in `fetch'
/srv/gitlab/lib/gitlab/repository_cache.rb:25:in `fetch'
/srv/gitlab/lib/gitlab/repository_cache_adapter.rb:178:in `block in cache_method_output_asymmetrically'
/srv/gitlab/lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
/srv/gitlab/lib/gitlab/repository_cache_adapter.rb:192:in `block in memoize_method_output'
/srv/gitlab/lib/gitlab/repository_cache_adapter.rb:201:in `no_repository_fallback'
/srv/gitlab/lib/gitlab/repository_cache_adapter.rb:191:in `memoize_method_output'
/srv/gitlab/lib/gitlab/repository_cache_adapter.rb:177:in `cache_method_output_asymmetrically'
/srv/gitlab/lib/gitlab/repository_cache_adapter.rb:83:in `block in cache_method_asymmetrically'
/srv/gitlab/app/models/repository.rb:534:in `empty?'
/srv/gitlab/app/models/repository.rb:351:in `expire_emptiness_caches'
/srv/gitlab/lib/backup/repository.rb:151:in `empty_repo?'
/srv/gitlab/lib/backup/repository.rb:35:in `block in dump'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.2/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.2/lib/active_record/relation/batches.rb:70:in `each'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.2/lib/active_record/relation/batches.rb:70:in `block in find_each'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.2/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.2/lib/active_record/relation/batches.rb:238:in `block in in_batches'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.2/lib/active_record/relation/batches.rb:222:in `loop'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.2/lib/active_record/relation/batches.rb:222:in `in_batches'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.2/lib/active_record/relation/batches.rb:135:in `find_in_batches'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.2/lib/active_record/relation/batches.rb:69:in `find_each'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.2/lib/active_record/querying.rb:21:in `find_each'
/srv/gitlab/lib/backup/repository.rb:17:in `dump'
/srv/gitlab/lib/tasks/gitlab/backup.rake:88:in `block (4 levels) in <top (required)>'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
/srv/gitlab/bin/bundle:3:in `load'
/srv/gitlab/bin/bundle:3:in `<main>'

Caused by:
GRPC::PermissionDenied: 7:permission denied
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/grpc-1.24.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:31:in `check_status'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/grpc-1.24.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:181:in `attach_status_results_and_complete_call'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/grpc-1.24.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:377:in `request_response'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/grpc-1.24.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:178:in `block in request_response'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/grpc-1.24.0-x86_64-linux/src/ruby/lib/grpc/generic/interceptors.rb:170:in `intercept!'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/grpc-1.24.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:177:in `request_response'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/grpc-1.24.0-x86_64-linux/src/ruby/lib/grpc/generic/service.rb:170:in `block (3 levels) in rpc_stub_class'
/srv/gitlab/lib/gitlab/gitaly_client.rb:168:in `execute'
/srv/gitlab/lib/gitlab/gitaly_client.rb:146:in `block in call'
/srv/gitlab/lib/gitlab/gitaly_client.rb:174:in `measure_timings'
/srv/gitlab/lib/gitlab/gitaly_client.rb:145:in `call'
/srv/gitlab/lib/gitlab/gitaly_client/repository_service.rb:99:in `has_local_branches?'
/srv/gitlab/lib/gitlab/git/repository.rb:1022:in `block in uncached_has_local_branches?'
/srv/gitlab/lib/gitlab/git/wraps_gitaly_errors.rb:7:in `wrapped_gitaly_errors'
/srv/gitlab/lib/gitlab/git/repository.rb:1021:in `uncached_has_local_branches?'
/srv/gitlab/lib/gitlab/git/repository.rb:161:in `block in has_local_branches?'
/srv/gitlab/lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
/srv/gitlab/lib/gitlab/git/repository.rb:160:in `has_local_branches?'
/srv/gitlab/app/models/repository.rb:560:in `has_visible_content?'
/srv/gitlab/lib/gitlab/repository_cache_adapter.rb:84:in `block (2 levels) in cache_method_asymmetrically'
/srv/gitlab/lib/gitlab/repository_cache.rb:44:in `fetch_without_caching_false'
/srv/gitlab/lib/gitlab/repository_cache_adapter.rb:179:in `block (2 levels) in cache_method_output_asymmetrically'
/srv/gitlab/lib/gitlab/null_request_store.rb:34:in `fetch'
/srv/gitlab/lib/gitlab/safe_request_store.rb:12:in `fetch'
/srv/gitlab/lib/gitlab/repository_cache.rb:25:in `fetch'
/srv/gitlab/lib/gitlab/repository_cache_adapter.rb:178:in `block in cache_method_output_asymmetrically'
/srv/gitlab/lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
/srv/gitlab/lib/gitlab/repository_cache_adapter.rb:192:in `block in memoize_method_output'
/srv/gitlab/lib/gitlab/repository_cache_adapter.rb:201:in `no_repository_fallback'
/srv/gitlab/lib/gitlab/repository_cache_adapter.rb:191:in `memoize_method_output'
/srv/gitlab/lib/gitlab/repository_cache_adapter.rb:177:in `cache_method_output_asymmetrically'
/srv/gitlab/lib/gitlab/repository_cache_adapter.rb:83:in `block in cache_method_asymmetrically'
/srv/gitlab/app/models/repository.rb:534:in `empty?'
/srv/gitlab/app/models/repository.rb:351:in `expire_emptiness_caches'
/srv/gitlab/lib/backup/repository.rb:151:in `empty_repo?'
/srv/gitlab/lib/backup/repository.rb:35:in `block in dump'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.2/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.2/lib/active_record/relation/batches.rb:70:in `each'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.2/lib/active_record/relation/batches.rb:70:in `block in find_each'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.2/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.2/lib/active_record/relation/batches.rb:238:in `block in in_batches'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.2/lib/active_record/relation/batches.rb:222:in `loop'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.2/lib/active_record/relation/batches.rb:222:in `in_batches'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.2/lib/active_record/relation/batches.rb:135:in `find_in_batches'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.2/lib/active_record/relation/batches.rb:69:in `find_each'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.2/lib/active_record/querying.rb:21:in `find_each'
/srv/gitlab/lib/backup/repository.rb:17:in `dump'
/srv/gitlab/lib/tasks/gitlab/backup.rake:88:in `block (4 levels) in <top (required)>'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
/srv/gitlab/bin/bundle:3:in `load'
/srv/gitlab/bin/bundle:3:in `<main>'
Tasks: TOP => gitlab:backup:repo:create
(See full trace by running task with --trace)
2020-03-17 08:59:12 +0000 -- Dumping repositories ...
 * molokai/development_dev (@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35) ... [WARNING] Executing 7:permission denied
Ignoring error on molokai/development_dev (@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35) - Failed to backup repo
[DONE]

We have already checked, that the gitlaly token is present on the task-runner pod as well as the rails ca-cert.

The backup sometimes concludes, but only, if someone deleted a currently running gitlab-task-runner-backup pod. This error is not related to archived projects (as the project molokai/development_dev was archived, then deleted, and then it failed on another, productive project).