PgBouncer deployment with terraform

In order to verify usage of PgBouncer in front of Postgres database PgBouncer included into terraform deployment. It uses separate machine with internal IP that should be used from Praefect instances and dedicated public IP that is marked as allowed in Cloud SQL instance. Cloud SQL authorized networks changed to dedicated public IP of the PgBouncer instance. It is not possible to cross-like them, as PgBouncer requires IP of Cloud SQL instance in setup (circular dependency). That is why dedicated IP is used for PgBouncer.
The output 'praefect_postgresql_ip' changed to 'praefect_pgbouncer_ip' and is a private IP of the PgBouncer instance that should be used instead of a public Cloud SQL instance to proxy SQL requests.

Closes: #2975 (closed)

Verification: Output of the terraform deployment:

gitaly_internal_ip = {
  "pstrokov-20200529-gitaly-1" = "10.150.0.125"
  "pstrokov-20200529-gitaly-2" = "10.150.0.107"
  "pstrokov-20200529-gitaly-3" = "10.150.0.123"
}
gitaly_ssh_ip = {
  "pstrokov-20200529-gitaly-1" = "35.245.127.77"
  "pstrokov-20200529-gitaly-2" = "35.188.253.69"
  "pstrokov-20200529-gitaly-3" = "35.194.92.18"
}
gitlab_external_ip = 35.245.136.225
gitlab_internal_ip = 10.150.0.13
praefect_internal_ip = {
  "pstrokov-20200529-praefect-1" = "10.150.0.122"
  "pstrokov-20200529-praefect-2" = "10.150.0.12"
  "pstrokov-20200529-praefect-3" = "10.150.0.124"
}
praefect_loadbalancer_ip = 10.150.0.126
praefect_pgbouncer_ip = 10.150.0.127
praefect_ssh_ip = {
  "pstrokov-20200529-praefect-1" = "35.236.217.72"
  "pstrokov-20200529-praefect-2" = "35.199.51.66"
  "pstrokov-20200529-praefect-3" = "34.86.132.127"
}

Setup done on the one of the praefect instances /etc/gitlab/gitlab.rb

postgresql['enable'] = false
redis['enable'] = false
nginx['enable'] = false
prometheus['enable'] = false
grafana['enable'] = false
puma['enable'] = false
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false
gitaly['enable'] = false

# Enable only the Praefect service
praefect['enable'] = true
praefect['listen_addr'] = '0.0.0.0:2305'
praefect['prometheus_listen_addr'] = '0.0.0.0:9652'
praefect['auth_token'] = 'PRAEFECT_EXTERNAL_TOKEN'
praefect['database_host'] = '10.150.0.127'
praefect['database_port'] = 5432
praefect['database_user'] = 'praefect'
praefect['database_password'] = 'PRAEFECT_SQL_PASSWORD'
praefect['database_dbname'] = 'praefect_production'
praefect['database_sslmode'] = 'disable'
# # Prevent database connections during 'gitlab-ctl reconfigure'
gitlab_rails['rake_cache_clear'] = false
gitlab_rails['auto_migrate'] = false

Check of SQL connection with 'sql-ping' by praefect:

root@pstrokov-20200529-praefect-1:~# sudo -u git /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-ping
WARN[0000] ignoring configured election strategy as failover is disabled  election_strategy=sql pid=25782
praefect sql-ping: OK
root@pstrokov-20200529-praefect-1:~# 

The log from pgbouncer cloud instance

pstrokov@pstrokov-20200529-pgbouncer ~ $ docker ps
CONTAINER ID        IMAGE                                                                COMMAND                  CREATED             STATUS              PORTS                    NAMES
1e2a94e3ee2b        gcr.io/stackdriver-agents/stackdriver-logging-agent:0.2-1.5.33-1-1   "/entrypoint.sh /usr…"   43 minutes ago      Up 43 minutes                                stackdriver-logging-agent
b3f64d592af8        edoburu/pgbouncer:latest                                             "/entrypoint.sh /usr…"   43 minutes ago      Up 43 minutes       0.0.0.0:5432->5432/tcp   pgbouncer
# log of pgbouncer
pstrokov@pstrokov-20200529-pgbouncer ~ $ docker logs b3f64d592af8
2020-07-24 06:44:28.059 UTC [1] LOG kernel file descriptor limit: 1048576 (hard: 1048576); max_client_conn: 100, max expected fd use: 112
2020-07-24 06:44:28.060 UTC [1] LOG listening on 0.0.0.0:5432
2020-07-24 06:44:28.060 UTC [1] LOG process up: PgBouncer 1.12.0, libevent 2.1.11-stable (epoll), adns: udns 0.4, tls: OpenSSL 1.1.1g  21 Apr 2020
2020-07-24 06:45:28.060 UTC [1] LOG stats: 0 xacts/s, 0 queries/s, in 0 B/s, out 0 B/s, xact 0 us, query 0 us, wait 0 us
2020-07-24 06:46:28.061 UTC [1] LOG stats: 0 xacts/s, 0 queries/s, in 0 B/s, out 0 B/s, xact 0 us, query 0 us, wait 0 us
...
2020-07-24 07:22:14.427 UTC [1] LOG C-0x55641d340a30: praefect_production/praefect@10.150.0.122:44394 login attempt: db=praefect_production user=praefect tls=no
2020-07-24 07:22:14.431 UTC [1] LOG S-0x55641d394500: praefect_production/praefect@35.245.146.166:5432 new connection to server (from 172.17.0.2:53180)
2020-07-24 07:22:15.691 UTC [1] LOG C-0x55641d340c40: praefect_production/praefect@10.150.0.122:44400 login attempt: db=praefect_production user=praefect tls=no
2020-07-24 07:22:15.886 UTC [1] LOG C-0x55641d340c40: praefect_production/praefect@10.150.0.122:44400 closing because: client close request (age=0s)
...
2020-07-24 07:24:28.095 UTC [1] LOG stats: 0 xacts/s, 0 queries/s, in 6 B/s, out 21 B/s, xact 1600 us, query 1600 us, wait 0 us
2020-07-24 07:24:32.280 UTC [1] LOG C-0x55641d340a30: praefect_production/praefect@10.150.0.122:44552 closing because: client close request (age=10s)
...
2020-07-24 07:28:24.445 UTC [1] LOG C-0x55641d340a30: praefect_production/praefect@10.150.0.122:44840 login attempt: db=praefect_production user=praefect tls=no
2020-07-24 07:28:28.098 UTC [1] LOG stats: 0 xacts/s, 0 queries/s, in 6 B/s, out 21 B/s, xact 1702 us, query 1702 us, wait 0 us
2020-07-24 07:28:34.450 UTC [1] LOG C-0x55641d340a30: praefect_production/praefect@10.150.0.122:44840 closing because: client close request (age=10s)
...