Support Puma key_password_command for SSL key decryption
What does this MR do?
This commit adds support for puma['ssl_key_password_command']
to
make it possible to store encrypted SSL keys on disk.
This requires Puma v6.3.0 (https://github.com/puma/puma/pull/3133) to work: gitlab!122200 (merged).
Related issues
Relates to #7799 (closed)
Testing locally
- Install
mkcert
(https://github.com/FiloSottile/mkcert/releases). - Run
mkcert -install
to install the local CA. - Run:
cd /var/opt/gitlab/gitlab-rails/etc
mkcert puma.localhost 127.0.0.1 ::1
chown git:git puma.localhost*
- Encrypt the key. I'll use
mypassword
as an example:
openssl rsa -aes256 -in puma.localhost+2-key.pem -out encrypted_puma.localhost+2-key.pem
Enter in the password twice to write the encrypted file.
- Add a script that will show the password. I'll use
/var/opt/gitlab/gitlab-rails/etc/puma-password
:
#!/bin/sh
echo "my password"
- Set the permissions:
chown git:git /var/opt/gitlab/gitlab-rails/etc/puma-password
chmod 770 /var/opt/gitlab/gitlab-rails/etc/puma-password
- In
/etc/gitlab/gitlab.rb
:
puma['ssl_listen'] = "127.0.0.1"
puma['ssl_port'] = 9111
puma['ssl_key_password_command'] = '/var/opt/gitlab/gitlab-rails/etc/puma-password'
puma['ssl_certificate'] = '/var/opt/gitlab/gitlab-rails/etc/puma.localhost+2.pem'
puma['ssl_certificate_key'] = '/var/opt/gitlab/gitlab-rails/etc/encrypted_puma.localhost+2-key.pem'
puma['listen'] = "" # Disable HTTP
- Run
gitlab-ctl reconfigure
.
Now try to use GitLab. Puma should start up successfully with SSL:
# ps -ef | grep "puma 6.3.0"
git 3549608 1804 32 17:21 ? 00:00:42 puma 6.3.0 (unix:///var/opt/gitlab/gitlab-rails/sockets/gitlab.socket,ssl://127.0.0.1:9111?cert=%2Fvar%2Fopt%2Fgitlab%2Fgitlab-rails%2Fetc%2Fpuma.localhost%2B2.pem&key=%2Fvar%2Fopt%2Fgitlab%2Fgitlab-rails%2Fetc%2Fpuma.localhost%2B2-key.pem&key_password_command=%2Fvar%2Fopt%2Fgitlab%2Fgitlab-rails%2Fetc%2Fpuma-password&verify_mode=none) [gitlab-puma-worker]
You can verify that Workhorse is also using this socket:
# ps -ef | grep gitlab-workhorse | grep auth
git 3548932 1795 0 17:20 ? 00:00:00 /opt/gitlab/embedded/bin/gitlab-workhorse -listenNetwork tcp -listenUmask 0 -listenAddr 0.0.0.0:1234 -authBackend https://127.0.0.1:9111 -documentRoot /opt/gitlab/embedded/service/gitlab-rails/public -pprofListenAddr -prometheusListenAddr localhost:9229 -secretPath /opt/gitlab/embedded/service/gitlab-rails/.gitlab_workhorse_secret -logFormat json -config config.toml
You can double check that SSL is being used here:
# openssl s_client -connect 127.0.0.1:9111 -showcerts -prexit < /dev/null | grep CN
Can't use SSL_get_servername
depth=1 O = mkcert development CA, OU = root@stanhu-gce, CN = mkcert root@stanhu-gce
verify return:1
depth=0 O = mkcert development certificate, OU = root@stanhu-gce
verify return:1
DONE
i:O = mkcert development CA, OU = root@stanhu-gce, CN = mkcert root@stanhu-gce
+l1IXKZxHQSUf4RUKnIwOd/8JedXu8/Jr/52H5f2V4FEQCNoFiSHSmIACRgezsSb
issuer=O = mkcert development CA, OU = root@stanhu-gce, CN = mkcert root@stanhu-gce
i:O = mkcert development CA, OU = root@stanhu-gce, CN = mkcert root@stanhu-gce
+l1IXKZxHQSUf4RUKnIwOd/8JedXu8/Jr/52H5f2V4FEQCNoFiSHSmIACRgezsSb
issuer=O = mkcert development CA, OU = root@stanhu-gce, CN = mkcert root@stanhu-gce
Checklist
See Definition of done.
For anything in this list which will not be completed, please provide a reason in the MR discussion.
Required
-
MR title and description are up to date, accurate, and descriptive. -
MR targeting the appropriate branch. -
Latest Merge Result pipeline is green. -
When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow.
For GitLab team members
If you don't have access to this, the reviewer should trigger these jobs for you during the review process.
-
The manual Trigger:ee-package
jobs have a green pipeline running against latest commit. -
If config/software
orconfig/patches
directories are changed, make sure thebuild-package-on-all-os
job within theTrigger:ee-package
downstream pipeline succeeded. -
If you are changing anything SSL related, then the Trigger:package:fips
manual job within theTrigger:ee-package
downstream pipeline must succeed. -
If CI configuration is changed, the branch must be pushed to dev.gitlab.org
to confirm regular branch builds aren't broken.
Expected (please provide an explanation if not completing)
-
Test plan indicating conditions for success has been posted and passes. -
Documentation created/updated. -
Tests added. -
Integration tests added to GitLab QA. -
Equivalent MR/issue for the GitLab Chart opened. -
Validate potential values for new configuration settings. Formats such as integer 10
, duration10s
, URIscheme://user:passwd@host:port
may require quotation or other special handling when rendered in a template and written to a configuration file.
Merge request reports
Activity
added devopssystems groupdistribution sectioncore platform labels
assigned to @stanhu
- A deleted user
added featureaddition typefeature labels
1 Message Please add the workflowready for review label once you think the MR is ready to for an initial review. Merge requests are handled according to the workflow documented in our handbook and should receive a response within the limit documented in our First-response SLO.
If you don't receive a response, please mention
@gitlab-org/distribution
, or one of our Project MaintainersIf needed, you can retry the
danger-review
job that generated this comment.Generated by
Dangeradded 1 commit
- 35117969 - Support Puma key_password_command for SSL key decryption
changed milestone to %16.1
mentioned in issue #7799 (closed)
marked the checklist item When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow. as completed
added workflowready for review label
marked the checklist item If CI configuration is changed, the branch must be pushed to
dev.gitlab.org
to confirm regular branch builds aren't broken. as completed Assigning myself as Reviewer as part of FY24Q2 Engage Distribution stable counterparts (gitlab-org/distribution/team-tasks#1251 - closed)I'll timebox the review until June 7 if that's fine as I need to read up on https://gitlab.com/gitlab-org/omnibus-gitlab/-/tree/master/doc/development!
requested review from @niskhakova
added workflowin review label and removed workflowready for review label
Testing
Truly appreciate detailed testing steps, thanks for documenting them
Learnt a lot- No
ssl_key_password_command
tested by QA pipelines - Tested locally
ssl_key_password_command
usage fromregistry.gitlab.com/gitlab-org/omnibus-gitlab/gitlab-ee:16.0.1-rfbranch.885598144.3291152d-0
image built in this branch - verified that SSL is used when following testing steps - Tested locally removing
ssl_
configs - verified that SSL is disabled
QA pipelines
Failures are unrelated and known:
- Failure in api/3_create/merge_request/push_opti... (gitlab#346425)
- Multiple failures due to tests trying to use new navigation without actually switching to it in the application, created QA failures in Omnibus pipelines due to new nav... (gitlab#414441 - closed) to track this
- No
@balasankarc could you please review as a maintainer?requested review from @balasankarc
mentioned in issue gitlab-org/distribution/team-tasks#1251 (closed)
mentioned in issue gitlab-org/quality/pipeline-triage#201 (closed)
mentioned in commit b9f27fc0
mentioned in merge request gitlab!122996 (merged)
mentioned in commit gitlab@4ba71660
mentioned in commit gitlab@889963f7
mentioned in commit gitlab@13ef050c
mentioned in commit gitlab@f580fb5c
mentioned in commit gitlab@e784f8bf
mentioned in commit gitlab@b325d8c3
mentioned in commit gitlab@37196005
added workflowstaging-canary label and removed workflowin review label
added workflowcanary label and removed workflowstaging-canary label
mentioned in commit gitlab@2190d855
added workflowstaging label and removed workflowcanary label
added workflowproduction label and removed workflowstaging label
mentioned in commit gitlab@05da19cc