[Feature flag] Rollout of `ci_jwt_signing_key`
What
Rollout and remove the :ci_jwt_signing_key
feature flag. This will switch from signing CI_JOB_JWT
tokens with the new dedicated RSA256 signing key instead of using the OIDC signing key as we do at the moment.
Owners
- Team: ~"group::release management"
- Most appropriate slack channel to reach out to:
#g_release-management
- Best individual to reach out to: @krasio
Expectations
What are we expecting to happen?
Once ci_jwt_signing_key
is enabled start signing CI_JOB_JWT
with dedicated signing key instead of using OIDC signing key.
What might happen if this goes wrong?
-
CI_JOB_JWT
generated for CI jobs can not be validated from 3rd parties. - (worst case) Error while generating
CI_JOB_JWT
breaks CI jobs for everyone, no matter are they using the JWTs or not. Unlike and already guarded pretty well with https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/build.rb#L1062-1064.
What can we monitor to detect problems with this?
- Errors when creating pipeline https://sentry.gitlab.net/gitlab/gitlabcom/?query=is%3Aunresolved+%22Ci%3A%3ACreatePipelineService%3A%3ACreateError%22
- Jobs failures on Runners (by Runner type) - https://dashboards.gitlab.net/d/000000159/ci?viewPanel=83&orgId=1
- Runners error 5m rate (by job&level) - https://dashboards.gitlab.net/d/000000159/ci?viewPanel=48&orgId=1
Beta groups/projects
If applicable, any groups/projects that are happy to have this feature turned on early. Some organizations may wish to test big changes they are interested in with a small subset of users ahead of time for example.
- Staging - https://staging.gitlab.com/krasio/ci-secrets/
- Production - https://gitlab.com/krasio/ci-secrets/
Roll Out Steps
-
Enable on staging ( /chatops run feature set feature_name true --staging
) -
Test on staging -
Ensure that documentation has been updated -
Enable on GitLab.com for individual groups/projects listed above and verify behaviour ( /chatops run feature set --project=gitlab-org/gitlab feature_name true
) -
Coordinate a time to enable the flag with #production
and#g_delivery
on slack. -
Announce on the issue an estimated time this will be enabled on GitLab.com -
Enable on GitLab.com by running chatops command in #production
(/chatops run feature set feature_name true
) -
Cross post chatops Slack command to #support_gitlab-com
(more guidance when this is necessary in the dev docs) and in your team channel -
Announce on the issue that the flag has been enabled -
Remove feature flag and add changelog entryMake feature flag enabled by default and add changelog entry -
After the flag removal is deployed, clean up the feature flag by running chatops command in #production
channel
Edited by Krasimir Angelov