Deadline Exceeded in Repository#delete_all_refs_except
A customer was testing my Bitbucket Server importer on a separate instance, and after the importer we noticed the Gitaly DeadlineExceeded message:
2018-07-31_19:50:14.13670 2018-07-31T19:50:14.135Z 28554 TID-owtthtnky WARN: Gitlab::Git::CommandError: 4:rpc error: code = DeadlineExceeded desc = context deadline exceeded
2018-07-31_19:50:14.13783 2018-07-31T19:50:14.135Z 28554 TID-owtthtnky WARN: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:1026:in `rescue in wrapped_gitaly_errors'
2018-07-31_19:50:14.13784 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:1019:in `wrapped_gitaly_errors'
2018-07-31_19:50:14.13784 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:260:in `delete_all_refs_except'
2018-07-31_19:50:14.13784 /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:467:in `method_missing'
2018-07-31_19:50:14.13784 /opt/gitlab/embedded/service/gitlab-rails/app/services/projects/after_import_service.rb:13:in `block in execute'
2018-07-31_19:50:14.13784 /opt/gitlab/embedded/service/gitlab-rails/app/services/projects/housekeeping_service.rb:29:in `execute'
2018-07-31_19:50:14.13784 /opt/gitlab/embedded/service/gitlab-rails/app/services/projects/after_import_service.rb:12:in `execute'
2018-07-31_19:50:14.13784 /opt/gitlab/embedded/service/gitlab-rails/app/models/project_import_state.rb:50:in `block (3 levels) in <class:ProjectImportState>'
2018-07-31_19:50:14.13785 /opt/gitlab/embedded/service/gitlab-rails/lib/after_commit_queue.rb:49:in `instance_eval'
2018-07-31_19:50:14.13785 /opt/gitlab/embedded/service/gitlab-rails/lib/after_commit_queue.rb:49:in `_run_after_commit_queue'
2018-07-31_19:50:14.13785 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:432:in `block in make_lambda'
2018-07-31_19:50:14.13785 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:263:in `block in simple'
2018-07-31_19:50:14.13785 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:506:in `block in call'
2018-07-31_19:50:14.13786 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:506:in `each'
2018-07-31_19:50:14.13786 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:506:in `call'
2018-07-31_19:50:14.13786 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
2018-07-31_19:50:14.13786 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:778:in `_run_commit_callbacks'
2018-07-31_19:50:14.13786 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/transactions.rb:314:in `committed!'
2018-07-31_19:50:14.13787 /opt/gitlab/embedded/service/gitlab-rails/config/initializers/forbid_sidekiq_in_transactions.rb:49:in `block in committed!'
2018-07-31_19:50:14.13787 /opt/gitlab/embedded/service/gitlab-rails/config/initializers/forbid_sidekiq_in_transactions.rb:11:in `skipping_transaction_check'
2018-07-31_19:50:14.13787 /opt/gitlab/embedded/service/gitlab-rails/config/initializers/forbid_sidekiq_in_transactions.rb:49:in `committed!'
2018-07-31_19:50:14.13787 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/transaction.rb:89:in `commit_records'
2018-07-31_19:50:14.13787 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/transaction.rb:153:in `commit'
2018-07-31_19:50:14.13787 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/transaction.rb:175:in `commit_transaction'
2018-07-31_19:50:14.13787 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/transaction.rb:194:in `within_new_transaction'
2018-07-31_19:50:14.13787 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
2018-07-31_19:50:14.13788 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/transactions.rb:220:in `transaction'
2018-07-31_19:50:14.13788 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:554:in `transaction'
2018-07-31_19:50:14.13788 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:1868:in `within_transaction'
2018-07-31_19:50:14.13788 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:170:in `within_transaction'
2018-07-31_19:50:14.13788 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:181:in `within_transaction'
2018-07-31_19:50:14.13788 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:62:in `perform'
2018-07-31_19:50:14.13789 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:528:in `around_save'
2018-07-31_19:50:14.13789 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:509:in `save'
2018-07-31_19:50:14.13789 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:154:in `block (2 levels) in run_actions'
2018-07-31_19:50:14.13789 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:154:in `each'
2018-07-31_19:50:14.13789 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:154:in `block in run_actions'
2018-07-31_19:50:14.13789 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:170:in `catch_exceptions'
2018-07-31_19:50:14.13790 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:148:in `run_actions'
2018-07-31_19:50:14.13790 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:60:in `perform'
2018-07-31_19:50:14.13790 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:163:in `perform'
2018-07-31_19:50:14.13791 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/state_machines-0.5.0/lib/state_machines/event.rb:155:in `fire'
2018-07-31_19:50:14.13791 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/state_machines-0.5.0/lib/state_machines/event.rb:219:in `block in add_actions'
2018-07-31_19:50:14.13791 /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:729:in `block (2 levels) in define_helper'
2018-07-31_19:50:14.13791 /opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:721:in `import_finish'
2018-07-31_19:50:14.13791 /opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:1629:in `after_import'