Support Puma key_password_command for SSL key decryption

What does this MR do?

This commit adds support for puma['ssl_key_password_command'] to make it possible to store encrypted SSL keys on disk.

This requires Puma v6.3.0 (https://github.com/puma/puma/pull/3133) to work: gitlab!122200 (merged).

Related issues

Relates to #7799 (closed)

Testing locally

1. Install mkcert (https://github.com/FiloSottile/mkcert/releases).
2. Run mkcert -install to install the local CA.
3. Run:

cd /var/opt/gitlab/gitlab-rails/etc
mkcert puma.localhost 127.0.0.1 ::1 
chown git:git puma.localhost*

4. Encrypt the key. I'll use mypassword as an example:

openssl rsa -aes256 -in puma.localhost+2-key.pem -out encrypted_puma.localhost+2-key.pem

Enter in the password twice to write the encrypted file.

5. Add a script that will show the password. I'll use /var/opt/gitlab/gitlab-rails/etc/puma-password:

#!/bin/sh
echo "my password"

6. Set the permissions:

chown git:git /var/opt/gitlab/gitlab-rails/etc/puma-password
chmod 770 /var/opt/gitlab/gitlab-rails/etc/puma-password

7. In /etc/gitlab/gitlab.rb:

puma['ssl_listen'] = "127.0.0.1"
puma['ssl_port'] = 9111
puma['ssl_key_password_command'] = '/var/opt/gitlab/gitlab-rails/etc/puma-password'
puma['ssl_certificate'] = '/var/opt/gitlab/gitlab-rails/etc/puma.localhost+2.pem'
puma['ssl_certificate_key'] = '/var/opt/gitlab/gitlab-rails/etc/encrypted_puma.localhost+2-key.pem'
puma['listen'] = "" # Disable HTTP

8. Run gitlab-ctl reconfigure.

Now try to use GitLab. Puma should start up successfully with SSL:

# ps -ef | grep "puma 6.3.0"
git      3549608    1804 32 17:21 ?        00:00:42 puma 6.3.0 (unix:///var/opt/gitlab/gitlab-rails/sockets/gitlab.socket,ssl://127.0.0.1:9111?cert=%2Fvar%2Fopt%2Fgitlab%2Fgitlab-rails%2Fetc%2Fpuma.localhost%2B2.pem&key=%2Fvar%2Fopt%2Fgitlab%2Fgitlab-rails%2Fetc%2Fpuma.localhost%2B2-key.pem&key_password_command=%2Fvar%2Fopt%2Fgitlab%2Fgitlab-rails%2Fetc%2Fpuma-password&verify_mode=none) [gitlab-puma-worker]

You can verify that Workhorse is also using this socket:

# ps -ef | grep gitlab-workhorse | grep auth
git      3548932    1795  0 17:20 ?        00:00:00 /opt/gitlab/embedded/bin/gitlab-workhorse -listenNetwork tcp -listenUmask 0 -listenAddr 0.0.0.0:1234 -authBackend https://127.0.0.1:9111 -documentRoot /opt/gitlab/embedded/service/gitlab-rails/public -pprofListenAddr  -prometheusListenAddr localhost:9229 -secretPath /opt/gitlab/embedded/service/gitlab-rails/.gitlab_workhorse_secret -logFormat json -config config.toml

You can double check that SSL is being used here:

# openssl s_client -connect 127.0.0.1:9111 -showcerts -prexit < /dev/null | grep CN
Can't use SSL_get_servername
depth=1 O = mkcert development CA, OU = root@stanhu-gce, CN = mkcert root@stanhu-gce
verify return:1
depth=0 O = mkcert development certificate, OU = root@stanhu-gce
verify return:1
DONE
   i:O = mkcert development CA, OU = root@stanhu-gce, CN = mkcert root@stanhu-gce
+l1IXKZxHQSUf4RUKnIwOd/8JedXu8/Jr/52H5f2V4FEQCNoFiSHSmIACRgezsSb
issuer=O = mkcert development CA, OU = root@stanhu-gce, CN = mkcert root@stanhu-gce
   i:O = mkcert development CA, OU = root@stanhu-gce, CN = mkcert root@stanhu-gce
+l1IXKZxHQSUf4RUKnIwOd/8JedXu8/Jr/52H5f2V4FEQCNoFiSHSmIACRgezsSb
issuer=O = mkcert development CA, OU = root@stanhu-gce, CN = mkcert root@stanhu-gce

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

• MR title and description are up to date, accurate, and descriptive.
• MR targeting the appropriate branch.
• Latest Merge Result pipeline is green.
• When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow.

For GitLab team members

If you don't have access to this, the reviewer should trigger these jobs for you during the review process.

• The manual Trigger:ee-package jobs have a green pipeline running against latest commit.
• If config/software or config/patches directories are changed, make sure the build-package-on-all-os job within the Trigger:ee-package downstream pipeline succeeded.
• If you are changing anything SSL related, then the Trigger:package:fips manual job within the Trigger:ee-package downstream pipeline must succeed.
• If CI configuration is changed, the branch must be pushed to dev.gitlab.org to confirm regular branch builds aren't broken.

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes.
• Documentation created/updated.
• Tests added.
• Integration tests added to GitLab QA.
• Equivalent MR/issue for the GitLab Chart opened.
• Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.

files/gitlab-config-template/gitlab.rb.template

@@ -1083,6 +1083,7 @@ external_url 'GENERATED_EXTERNAL_URL'
 # puma['ssl_certificate_key'] = nil
 # puma['ssl_client_certificate'] = nil
 # puma['ssl_cipher_filter'] = nil
+# puma['ssl_key_password_command'] = nil
 # puma['ssl_verify_mode'] = 'none'
 
 # puma['pidfile'] = '/opt/gitlab/var/puma/puma.pid'