Change SIGTERM behaviour for PgBouncer

What does this MR do?

Change SIGTERM behaviour for PgBouncer

This commit provides a workaround for the following PgBouncer breaking change:

Starting from PgBouncer 1.23, SIGTERM doesn't cause immediate shutdown of the PgBouncer process anymore. It now does a “super safe shutdown”: waiting for all clients to disconnect before shutting down. This is a minor breaking change. If you relied on the old behaviour of SIGTERM you should now use SIGQUIT.

See: https://www.pgbouncer.org/changelog.html#pgbouncer-123x

We now intercept TERM signal in Runit control, send the TERM first, wait up to SVWAIT (default 7) seconds, if the process is still alive we send QUIT signal for immediate shutdown.

Related issues

Closes #8617 (closed)

Test plan

Setup

1. Setup a pgbouncer node using the Docker image or Ubuntu package of this MR with the following gitlab.rb:

   roles ['pgbouncer_role']
   
   pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul)
   pgbouncer['users'] = {
     'gitlab-consul': {
       password: 'secret',
     },
     'pgbouncer': {
       password: 'secret',
     }
   }

   Alternatively, you can use https://gitlab.com/pursultani/gitlab-ha-test-kit/

   export TEST_TAG="..."
   source ./script.sh
   configure_pgb
   # Enter password

Testing with no open connection

1. Try gitlab-ctl stop pgbouncer. After a few seconds the process will terminate. You can check the pgbouncer logs to see the traces of the control script.

Testing with an open connection

1. Connect to PgBouncer console and leave it open:

   /opt/gitlab/embedded/bin/psql -h localhost -p 6432 -U pgbouncer -d gitlabhq_production

2. Try gitlab-ctl stop pgbouncer. After a few seconds the process will terminate. You can check the pgbouncer logs to see the traces of the control script.

Does it work?

You can remove (or move) the control script in /opt/gitlab/sv/pgbouncer/control/t and try "Testing with an open connection". gitlab-ctl stop pgbouncer times out.

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

• MR title and description are up to date, accurate, and descriptive.
• MR targeting the appropriate branch.
• Latest Merge Result pipeline is green.
• When ready for review, MR is labeled workflowready for review per the Distribution MR workflow.

For GitLab team members

If you don't have access to this, the reviewer should trigger these jobs for you during the review process.

• The manual Trigger:ee-package jobs have a green pipeline running against latest commit.
• If config/software or config/patches directories are changed, make sure the build-package-on-all-os job within the Trigger:ee-package downstream pipeline succeeded.
• If you are changing anything SSL related, then the Trigger:package:fips manual job within the Trigger:ee-package downstream pipeline must succeed.
• If CI configuration is changed, the branch must be pushed to dev.gitlab.org to confirm regular branch builds aren't broken.

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes.
• Documentation created/updated.
• Tests added.
• Integration tests added to GitLab QA.
• Equivalent MR/issue for the GitLab Chart opened.
• Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.