[Feature flag] Rollout of `ci_jwt_signing_key`
Rollout and remove the
:ci_jwt_signing_key feature flag. This will switch from signing
CI_JOB_JWT tokens with the new dedicated RSA256 signing key instead of using the OIDC signing key as we do at the moment.
- Team: ~"group::release management"
- Most appropriate slack channel to reach out to:
- Best individual to reach out to: @krasio
What are we expecting to happen?
ci_jwt_signing_key is enabled start signing
CI_JOB_JWT with dedicated signing key instead of using OIDC signing key.
What might happen if this goes wrong?
CI_JOB_JWTgenerated for CI jobs can not be validated from 3rd parties.
- (worst case) Error while generating
CI_JOB_JWTbreaks CI jobs for everyone, no matter are they using the JWTs or not. Unlike and already guarded pretty well with https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/build.rb#L1062-1064.
What can we monitor to detect problems with this?
- Errors when creating pipeline https://sentry.gitlab.net/gitlab/gitlabcom/?query=is%3Aunresolved+%22Ci%3A%3ACreatePipelineService%3A%3ACreateError%22
- Jobs failures on Runners (by Runner type) - https://dashboards.gitlab.net/d/000000159/ci?viewPanel=83&orgId=1
- Runners error 5m rate (by job&level) - https://dashboards.gitlab.net/d/000000159/ci?viewPanel=48&orgId=1
If applicable, any groups/projects that are happy to have this feature turned on early. Some organizations may wish to test big changes they are interested in with a small subset of users ahead of time for example.
- Staging - https://staging.gitlab.com/krasio/ci-secrets/
- Production - https://gitlab.com/krasio/ci-secrets/
Roll Out Steps
Enable on staging (
/chatops run feature set feature_name true --staging)
Test on staging
Ensure that documentation has been updated
Enable on GitLab.com for individual groups/projects listed above and verify behaviour (
/chatops run feature set --project=gitlab-org/gitlab feature_name true)
Coordinate a time to enable the flag with
Announce on the issue an estimated time this will be enabled on GitLab.com
Enable on GitLab.com by running chatops command in
/chatops run feature set feature_name true)
Cross post chatops Slack command to
#support_gitlab-com(more guidance when this is necessary in the dev docs) and in your team channel
Announce on the issue that the flag has been enabled
Remove feature flag and add changelog entryMake feature flag enabled by default and add changelog entry
After the flag removal is deployed, clean up the feature flag by running chatops command in