The value of ci_jwt_signing_key does not regenerate when lost/removed
Summary
In the documented steps to remove secrets when their keys are lost (such as during restores, etc.) the instructions ask users to run the following:
(From https://docs.gitlab.com/ee/raketasks/backup_restore.html#reset-runner-registration-tokens)
UPDATE application_settings SET encrypted_ci_jwt_signing_key = null;
Unlike the other columns similarly touched, the value of encrypted_ci_jwt_signing_key is never set automatically ever again (no ensure
tokens method, etc.)
Steps to reproduce
- Run the above doc-suggested SQL statement
- Try to use
CI_JOB_JWT
from inside a new job - fails
Example Project
This affects only self-managed instances, and is an instance-wide issue not limited to one project.
What is the current bug behavior?
The encrypted_ci_jwt_signing_key is never regenerated
What is the expected correct behavior?
The encrypted_ci_jwt_signing_key should get auto-regenerated from the persisted value in gitlab-secrets.json
(if found) or a new one, at reconfigure or a runtime step.
Relevant logs and/or screenshots
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)
Possible fixes
If there exists an existing value in the preserved /etc/gitlab/gitlab-secrets.json
file (or equivalent), copy its string-form value that is printed by running the below:
cat /etc/gitlab/gitlab-secrets.json | jq '.gitlab_rails.ci_jwt_signing_key'
Then run within the GitLab Rails Console:
KEY = <PASTE COPIED STRING HERE>
ApplicationSetting.find_each do |application_setting|
application_setting.update(ci_jwt_signing_key: KEY)
end
If ci_jwt_signing_key does not exist in the JSON, generate a new one, print it (and copy the value for later), running the following console command:
KEY = OpenSSL::PKey::RSA.new(2048).to_pem
# Print to store back into the secrets JSON file (to persist it)
pp KEY
ApplicationSetting.find_each do |application_setting|
application_setting.update(ci_jwt_signing_key: KEY)
end
Then save the copied value in the gitlab-secrets.json
file as a sub-key-value in the following nested area:
{
[…]
"gitlab_rails": {
"ci_jwt_signing_key": <PASTE STRING FORM OF PEM HERE>
[…]
}
[…]
}