[Feature flag] Rollout of `ci_jwt_signing_key`

What

Rollout and remove the :ci_jwt_signing_key feature flag. This will switch from signing CI_JOB_JWT tokens with the new dedicated RSA256 signing key instead of using the OIDC signing key as we do at the moment.

Owners

• Team: ~"group::release management"
• Most appropriate slack channel to reach out to: #g_release-management
• Best individual to reach out to: @krasio

Expectations

What are we expecting to happen?

Once ci_jwt_signing_key is enabled start signing CI_JOB_JWT with dedicated signing key instead of using OIDC signing key.

What might happen if this goes wrong?

1. CI_JOB_JWT generated for CI jobs can not be validated from 3rd parties.
2. (worst case) Error while generating CI_JOB_JWT breaks CI jobs for everyone, no matter are they using the JWTs or not. Unlike and already guarded pretty well with https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/build.rb#L1062-1064.

What can we monitor to detect problems with this?

• Errors when creating pipeline https://sentry.gitlab.net/gitlab/gitlabcom/?query=is%3Aunresolved+%22Ci%3A%3ACreatePipelineService%3A%3ACreateError%22
• Jobs failures on Runners (by Runner type) - https://dashboards.gitlab.net/d/000000159/ci?viewPanel=83&orgId=1
• Runners error 5m rate (by job&level) - https://dashboards.gitlab.net/d/000000159/ci?viewPanel=48&orgId=1

Beta groups/projects

If applicable, any groups/projects that are happy to have this feature turned on early. Some organizations may wish to test big changes they are interested in with a small subset of users ahead of time for example.

• Staging - https://staging.gitlab.com/krasio/ci-secrets/
• Production - https://gitlab.com/krasio/ci-secrets/

Roll Out Steps

• Enable on staging (/chatops run feature set feature_name true --staging)
• Test on staging
• Ensure that documentation has been updated
• Enable on GitLab.com for individual groups/projects listed above and verify behaviour (/chatops run feature set --project=gitlab-org/gitlab feature_name true)
• Coordinate a time to enable the flag with #production and #g_delivery on slack.
• Announce on the issue an estimated time this will be enabled on GitLab.com
• Enable on GitLab.com by running chatops command in #production (/chatops run feature set feature_name true)
• Cross post chatops Slack command to #support_gitlab-com (more guidance when this is necessary in the dev docs) and in your team channel
• Announce on the issue that the flag has been enabled
• Remove feature flag and add changelog entry Make feature flag enabled by default and add changelog entry
• After the flag removal is deployed, clean up the feature flag by running chatops command in #production channel