Backup process fails with DeadlineExceeded for emptiness check
Zendesk: https://gitlab.zendesk.com/agent/tickets/143470/events (internal)
Most DeadlineExceeded
errors during backup are caught, warned/logged, and the backup process continues. Here's an example:
* group/project ... [WARNING] Executing 4:Deadline Exceeded
Ignoring error on group/project - Failed to backup repo
However, one customer ran into a problem where the exists?
or emptiness check on a repo resulted in a DeadlineExceeded
, and this error is not caught and causes a fatal error. We're still investigating why a simple check like this would cause deadline exceeded but it is probably a corrupt repo or something.
During the actual backup of a repo, we catch errors and log them - https://gitlab.com/gitlab-org/gitlab/blob/8833618937b8c885d505af6302a9807e6c22cdfe/lib/backup/repository.rb#L50
However, we don't wrap the empty repo check - https://gitlab.com/gitlab-org/gitlab/blob/8833618937b8c885d505af6302a9807e6c22cdfe/lib/backup/repository.rb#L26 (repo) and https://gitlab.com/gitlab-org/gitlab/blob/8833618937b8c885d505af6302a9807e6c22cdfe/lib/backup/repository.rb#L35 (wiki). This results in the DeadlineExceeded failing the whole backup.
rake aborted!
GRPC::DeadlineExceeded: 4:Deadline Exceeded
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:154:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:19:in `exists?'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:89:in `exists?'
/opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:532:in `exists?'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:84:in `block (2 levels) in cache_method_asymmetrically'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache.rb:43:in `fetch_without_caching_false'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:174:in `block (2 levels) in cache_method_output_asymmetrically'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/null_request_store.rb:34:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/safe_request_store.rb:12:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache.rb:24:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:173:in `block in cache_method_output_asymmetrically'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:187:in `block in memoize_method_output'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:196:in `no_repository_fallback'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:186:in `memoize_method_output'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:172:in `cache_method_output_asymmetrically'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:83:in `block in cache_method_asymmetrically'
/opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:540:in `empty?'
/opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:359:in `expire_emptiness_caches'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:150:in `empty_repo?'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:35:in `block in dump'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:17:in `dump'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:88:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:11:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:repo:create
(See full trace by running task with --trace)