gitlab_rails['ci_jwt_signing_key'] setting has no effect
Summary
Setting the gitlab_rails['ci_jwt_signing_key']
value in gitlab.rb
or the GITLAB_OMNIBUS_CONFIG
environment variable has no effect, and a new private key is always generated during deployment.
Steps to reproduce
Prerequisites:
Generate an RSA private key and create a GitLab Docker container, setting the gitlab_rails['ci_jwt_signing_key']
value to the generated private key as shown:
openssl genrsa -out key.pem
container="$(docker run -p 80:80 -d -e GITLAB_OMNIBUS_CONFIG="gitlab_rails['ci_jwt_signing_key'] = \"$(cat key.pem)\"" gitlab/gitlab-ce)"
What is the current bug behavior?
Attempting to validate a JWT signed with our generated key.pem
against the GitLab instance's /-/jwks
endpoint as shown below fails with cannot find key with kid ... on gitlab.jwks
:
curl http://localhost/-/jwks > gitlab.jwks
step crypto jwt sign -iss 'test' --aud 'gitlab' --sub 'test' --exp "$(date -d 'tomorrow' +%s)" --key key.pem | step crypto jwt verify --jwks gitlab.jwks --iss 'test' --aud 'gitlab'
What is the expected correct behavior?
The above command should succeed, since GitLab's /-/jwks
endpoint should contain a public key corresponding to our generated key.pem
private key.
Notably, if we inspect the container's contents, we can see that our generated key.pem
was successfully placed into /etc/gitlab/gitlab-secrets.json
and /var/opt/gitlab/gitlab-rails/etc/secrets.yml
:
cat key.pem # for comparison
docker exec $container cat /etc/gitlab/gitlab-secrets.json | jq -r '.gitlab_rails.ci_jwt_signing_key'
docker exec $container cat /var/opt/gitlab/gitlab-rails/etc/secrets.yml | yq -r '.production.ci_jwt_signing_key'
However, the actual setting value in GitLab does not match our generated key.pem
:
echo 'puts Gitlab::CurrentSettings.ci_jwt_signing_key' | docker exec -i $container gitlab-rails console | tail -n +10 | head -n -2 > actual.pem
cat actual.pem # does not match key.pem
If we try to validate a JWT signed with this actual.pem
against the GitLab instance's /-/jwks
endpoint, it succeeds as expected:
step crypto jwt sign -iss 'test' --aud 'gitlab' --sub 'test' --exp "$(date -d 'tomorrow' +%s)" --key actual.pem | step crypto jwt verify --jwks gitlab.jwks --iss 'test' --aud 'gitlab'
Why does this happen?
Looking through the project histories, it seems that the ci_jwt_signing_key
setting was initially implemented in both omnibus-gitlab
and gitlab
as a Rails secret, in the following commits:
The change to gitlab
was later rolled back though, due to upgrade issues.
Eventually, the decision was made to store this key in an encrypted application setting rather than a Rails secret, and this implementation was merged. As part of this implementation, a database migration was added that will always generate a fresh JWT signing key, ignoring any value of gitlab_rails['ci_jwt_signing_key']
.
However, omnibus-gitlab
was never updated to set the ci_jwt_signing_key
application setting based on the value of gitlab_rails['ci_jwt_signing_key']
set in gitlab.rb
, and instead continues to populate the ci_jwt_signing_key
Rails secret, which is not used by GitLab.
How should this be fixed?
Given the reasoning behind changing ci_jwt_signing_key
from a Rails secret to an encrypted application setting, it seems like the correct fix is to update omnibus-gitlab
to set the application setting to the gitlab_rails['ci_jwt_signing_key']
value present in gitlab.rb
. I'm unable to find any precedent for populating application settings from gitlab.rb
in omnibus-gitlab
, so I'm not sure how complex this would be.
How can this be worked around?
Create a post-deployment database migration that populates the ci_jwt_signing_key
application setting from an environment variable, then mount it into the Docker container and populate the environment variable as shown:
cat << EOF > 100_ci_jwt_signing_key.rb
Gitlab::CurrentSettings.current_application_settings.tap do |settings|
settings.ci_jwt_signing_key = ENV['GITLAB_CI_JWT_SIGNING_KEY']
end.save
EOF
openssl genrsa -out key.pem
container="$(docker run -p 80:80 -d -v 100_ci_jwt_signing_key.rb:/opt/gitlab/embedded/service/gitlab-rails/db/fixtures/production/100_ci_jwt_signing_key.rb -e GITLAB_CI_JWT_SIGNING_KEY="$(cat key.pem)" gitlab/gitlab-ce)"
Repeat the provided validation steps and observe that the bug behavior no longer reproduces.
Details of package version
$ dpkg-query -l "gitlab-*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-===========================================================
ii gitlab-ce 16.8.2-ce.0 amd64 GitLab Community Edition (including NGINX, Postgres, Redis)
un gitlab-ee <none> <none> (no description available)
un gitlab-fips <none> <none> (no description available)
Environment details
- Operating System:
Ubuntu 22.04.3 LTS
- Installation Target, remove incorrect values:
- Other:
docker
- Other:
- Installation Type, remove incorrect values:
- New Installation
- Is there any other software running on the machine: No
- Is this a single or multiple node installation?
- Single
- Resources
- CPU:
8 cores
- Memory total:
8 GB
- CPU: