"template/generic-monograph" did not exist on "main"
Gitlab CE 11.1.1/2 Backup faile on large repos since update to 11.1
Since we updated to version 11.1 (now running gitlab-ce-11.1.2-ce.0.el7.x86_64) most of the backup runs (gitlab-rake gitlab:backup:create STRATEGY=copy
) fail.
The backup randomly fails at one of our larger repositories ( > 1GB ). It is not always the same repo, but always one of them.
Sometimes there is only a warning:
...
* group/LargeRepo ... [WARNING] Executing 4:Deadline Exceeded
Ignoring error on papers/MinimalPatterns - Failed to backup repo
[DONE]
[SKIPPED] Wiki
...
Sometimes the backup process crashes:
* group/LargeRepo ... rake aborted!
GRPC::Core::CallError: call#run_batch failed somehow
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:348:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:348:in `block in gitaly_fetch_stream_to_file'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:347:in `open'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:347:in `gitaly_fetch_stream_to_file'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:201:in `create_bundle'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:67:in `backup_project_gitaly'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:53:in `block in backup_project'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:289:in `block (2 levels) in migrate'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:329:in `allow_n_plus_1_calls'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:282:in `block in migrate'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/influx_db.rb:98:in `measure'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:280:in `migrate'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:285:in `gitaly_migrate'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:51:in `backup_project'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:27:in `block in dump'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:17:in `dump'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:80:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:11:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:repo:create
(See full trace by running task with --trace)
Before the update we were running 11.0.4 the backups were working reliable.