Log when database load balancer retries read_write

What does this MR do and why?

This MR adds some additional logging when read_write block is retried by the DB load balancer because of connection error.

This is needed in order to give us more visibility with 2022-10-05: Rollout PREVENT_LOAD_BALANCER_RETRI... (gitlab-com/gl-infra/production#7836 - closed).

How to set up and validate locally

Open the rails and psql console.

In the rails console, execute the following

ApplicationRecord.connection.execute('select pg_sleep(5)')

During the 5 seconds of sleep, run the following sql to terminate the sleeping connection (and all other connections to your local database):

SELECT pg_terminate_backend(pg_stat_activity.pid)
FROM pg_stat_activity
WHERE pg_stat_activity.datname = current_database() 
AND pid <> pg_backend_pid() AND query LIKE '%pg_sleep%';

In log/database_load_balancing.log you should see the following

{"severity":"WARN","time":"2022-11-11T03:16:54.929Z","correlation_id":null,"event":"read_write_retry","message":"A read_write block was retried because of connection error"}

You can also try it with transaction

ApplicationRecord.transaction { ApplicationRecord.connection.execute('select pg_sleep(5)') }

or with

PREVENT_LOAD_BALANCER_RETRIES_IN_TRANSACTION=1 be rails c

All should result in event read_write_retry logged.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

Related to #220242 (closed)