Consider using a proxy-based solution instead of EE load balancing code
Right now we're using the EE load balancing code (https://docs.gitlab.com/ee/administration/database_load_balancing.html) to distribute load across multiple replicas.
This has served us well, but during the incident in https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7543, SREs had trouble verifying whether this load balancing was doing the right thing.
A number of people have proposed that we switch to a solution based on HAProxy/Google load balancer/etc. that might be able to make smarter decisions about how to distribute load across multiple PgBouncer instances (e.g. based on ping times, etc.).
Right now our database configuration looks like:
load_balancing: {"hosts":[],"discover":{"record":"replica.patroni.service.consul.","nameserver":"127.0.0.1","port":8600,"use_tcp":true}}
Could we do what we do with the primary and just point it to a single hostname, turn off service discovery, and allow Consul/HAProxy/Google load balancer direct traffic to the right host? Are there issues with ActiveRecord connections needing to track connections for different hosts?