SQL queries running in Kubernetes are failing back to primary with "No secondaries were available, using primary instead"
When no secondary postgres servers are available, the load balancer logic will fail back to using the primary, with the following log message:
"No secondaries were available, using primary instead"
LoadBalancing::Logger.warn(
event: :no_secondaries_available,
message: 'No secondaries were available, using primary instead',
conflict_retried: conflict_retried,
host_list_length: @host_list.length
)
This is not ideal, since additional load will be generated on the primary.
As part of the rapid action in &5434 (closed), I've noticed that kubernetes provisioned services, primarily the git
fleet appears to be doing this, sometimes in bursts. Nearly all of these warnings are coming from the kubernetes hosts, and predominantly git
(which makes the most database queries out of the kubernetes rails fleet thus far).
I wonder consider at least understanding this problem a problem for further kubernetes rollout.
https://log.gprd.gitlab.net/goto/29ab6ec5ec59dc2654116f4a5625b8cd
Tracking for the last 24 hours https://log.gprd.gitlab.net/goto/0ef478224a698958bf53d0e61600d833
https://log.gprd.gitlab.net/goto/1c33ba387aa7b4e265a814df03ec8e29
cc @amyphillips @skarbek @marin @hphilipps @craig-gomes @abrandl @pbair @iroussos