Gitaly: Call dropped by load balancing policy
This appears to be affecting projects on nfs-09.
- https://sentry.gitlap.com/gitlab/gitlabcom/issues/108600/
- https://sentry.gitlap.com/gitlab/gitlabcom/issues/108638/
Incidents of 503 status errors over the last 24 hours:
https://log.gitlap.com/goto/d44b960fb3c394e691b86487a4c220b1
GRPC::Core::CallError: call#run_batch failed somehow
lib/gitlab/gitaly_client.rb:78:in `call'
stub(service, storage).__send__(rpc, request, metadata) # rubocop:disable GitlabSecurity/PublicSend
lib/gitlab/gitaly_client/ref_service.rb:39:in `tag_names'
response = GitalyClient.call(@storage, :ref_service, :find_all_tag_names, request)
lib/gitlab/git/repository.rb:230:in `block in tag_names'
gitaly_ref_client.tag_names
lib/gitlab/gitaly_client.rb:153:in `block (2 levels) in migrate'
yield is_enabled
lib/gitlab/gitaly_client.rb:193:in `allow_n_plus_1_calls'
yield
...
(156 additional frame(s) were not displayed)
GRPC::Unknown: 2:Call dropped by load balancing policy
lib/gitlab/gitaly_client.rb:78:in `call'
stub(service, storage).__send__(rpc, request, metadata) # rubocop:disable GitlabSecurity/PublicSend
lib/gitlab/gitaly_client/ref_service.rb:39:in `tag_names'
response = GitalyClient.call(@storage, :ref_service, :find_all_tag_names, request)
lib/gitlab/git/repository.rb:230:in `block in tag_names'
gitaly_ref_client.tag_names
lib/gitlab/gitaly_client.rb:153:in `block (2 levels) in migrate'
yield is_enabled
lib/gitlab/gitaly_client.rb:193:in `allow_n_plus_1_calls'
yield
...
(159 additional frame(s) were not displayed)
Gitlab::Git::CommandError: 2:Call dropped by load balancing policy
lib/gitlab/git/repository.rb:1126:in `rescue in gitaly_migrate'
raise CommandError.new(e)
lib/gitlab/git/repository.rb:1122:in `gitaly_migrate'
Gitlab::GitalyClient.migrate(method, status: status, &block)
lib/gitlab/git/repository.rb:228:in `tag_names'
gitaly_migrate(:tag_names) do |is_enabled|
app/models/repository.rb:530:in `tag_names'
delegate :tag_names, to: :raw_repository
lib/gitlab/metrics/instrumentation.rb:156:in `block in _uncached_tag_names'
trans.method_call_for(#{label.to_sym.inspect}).measure { super }
...
(145 additional frame(s) were not displayed)
Gitlab::Git::CommandError: 2:Call dropped by load balancing policy
Edited by Stan Hu