Explore adding failsafe check to prevent DB Migrations from running through PgBouncer
Summary
Spinning this out from #7716 as something that is hopefully more tangible and possible in the short term.
A recent customer escalation occurred due to a misfire of DB migrations through PgBouncer incorrectly. This was due to a bug in GET but generally this is arguably a brittle area with little safeguards in place and an easy to make mistake. Of note migrations don't always fail when this happens with the lock issue PgBouncer can create being sporadic.
While the issue is now fixed in GET it would be prudent to prevent it happening in the first place to benefit all users. If it's technically possible the relevant gitlab-ctl
commands should check if they're running against a PgBouncer and stop accordingly.
Proposal
The gitlab-ctl reconfigure
and gitlab-rake db:migrate
commands (and variants) check if they are connecting through a PgBouncer and stop accordingly with guidance given to change the config to be direct to DB.