### Deprecation Summary The Job Token format is being moved from a legacy string to a JWT format to support various requirements in the product. As a result, the Runner with AWS Fargate Custom Executor driver on version 0.5.0 or earlier is incompatible with the new CI JWT job token format. #### Documentation - Deprecation notice: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/178980+ - Migration guidelines: [Use toggle to switch to legacy format](https://docs.gitlab.com/ci/jobs/ci_job_token/#use-legacy-format-for-cicd-tokens). - [Troubleshooting base64 encoding](https://docs.gitlab.com/ci/jobs/ci_job_token/#invalid-character-n-in-string-literal-error-with-base64-encoding) #### Product Usage The job token is generated for every job and is used to access the GitLab API. The token is originally as encoded string and will be switching to the JWT format to support various product requirements inlcuding: * Embed Composite Identity inside CI_JOB_TOKEN * Encoding granular permissions for CI_JOB_TOKEN * https://gitlab.com/gitlab-org/gitlab/-/issues/489855+ * CI now maintains 2 versions of the CI_JOB_TOKEN and we need to remove the maintenance burden. 1. Existing/legacy encrypted string. [This is generated](https://gitlab.com/gitlab-org/gitlab/-/blob/3faf3ee65cf0189030109ac9156c5d81d13f6f7e/app/models/ci/build.rb#L321) when the job goes into `pending` state and persisted in the database. Then, when the job completes we [remove it](https://gitlab.com/gitlab-org/gitlab/-/blob/3faf3ee65cf0189030109ac9156c5d81d13f6f7e/app/workers/ci/build_finished_worker.rb#L43). 2. New JWT implementation. This is generated on the fly when we send the token to the runner. ### Known Issues 1. **AWS Fargate Driver Compatibility**: GitLab Runner's AWS Fargate Driver versions 0.5.0 and earlier are incompatible with the JWT standard. If you use AWS Fargate custom executor driver, please upgrade to version 0.5.1 or later. Migration instructions are available in our documentation. 2. **Base64 Encoding**: The JWT standard tokens are longer, which breaks the `echo $CI_JOB_TOKEN | base64` command if used in your CI/CD configurations. Please update to `echo $CI_JOB_TOKEN | base64 -w0` instead. ### Breaking Change? Does this deprecation contain a breaking change? `Yes` ### Affected Customers Who is affected by this deprecation: GitLab.com users, Self-managed users, or Dedicated users? (choose all that apply) - [x] GitLab.com - [x] Self-managed - [x] Dedicated What pricing tiers are impacted? - [x] GitLab Free - [x] GitLab Premium - [x] GitLab Ultimate ### Deprecation Milestone This deprecation will be announced in milestone: `17.9` ### Planned Removal Milestone The feature / functionality will be removed in milestone: `19.0` ### Links ### Checklists ### Timeline #### Rollout Plan - [ ] DRI Engineers: `@engineer(s)` * DRI Engineering Manager: @jayswain ft. @ajaythomasinc ##### Development Tasks * [x] [Create ability to switch JWT token between legacy and new format.](https://gitlab.com/gitlab-org/gitlab/-/work_items/514860) * [x] Identify customers who use Fargate driver to communicate upgrade path. (https://gitlab.com/gitlab-org/gitlab/-/issues/517754, https://gitlab.com/gitlab-org/gitlab/-/issues/517753) * [x] Update documentation #### Communication Plan DRI Product Manager: @jrandazzo ##### Internal communication * [x] [Merge deprecation entry](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/178980) * [x] Create troubleshooting guide ##### External Communication * [x] [Create communication memo on upgrade](https://docs.google.com/document/d/1zFi5JJQ6kVqVbUoiCykTklq7y6MJV0fUdPefv2u_KfA/edit?tab=t.0) * [x] Communication memo reviewed and approved by comms team * [ ] Review customers and notify attached AEs and CSMs * [ ] Send communication to customers #### Approvals - [ ] Product Manager `@PM` - [ ] Engineering Manager `@EM` - [ ] Senior Engineering Manager / Director `@senior-eng-leader` - [ ] Group / Director of Product Management `@senior-product-leader` - [ ] Product / Eng Leaders in the `CPO` or `CTO` organizations, as applicable (optional - depends on scope of change) References - [Deprecations, removals, and breaking changes](https://handbook.gitlab.com/handbook/marketing/blog/release-posts/#deprecations-removals-and-breaking-changes) - [Deprecation guidelines](https://docs.gitlab.com/ee/development/deprecation_guidelines/) - [Deprecations and removals doc styleguide](https://docs.gitlab.com/ee/development/documentation/styleguide/deprecations_and_removals) - [REST API Deprecations](https://docs.gitlab.com/ee/development/documentation/restful_api_styleguide.html#deprecations) and [REST API breaking changes](https://docs.gitlab.com/ee/development/api_styleguide.html#breaking-changes). - [GraphQL Deprecations](https://docs.gitlab.com/ee/development/api_graphql_styleguide.html#deprecating-schema-items) and [GraphQL API breaking changes](https://docs.gitlab.com/ee/development/api_graphql_styleguide.html#breaking-changes). - [GitLab release and maintenance policy](https://docs.gitlab.com/ee/policy/maintenance.html) - Videos :tv: - [How to deprecate and remove features in GitLab releases](https://youtu.be/9gy7tg94j7s) - [Review of GitLab deprecations and removals policy & Runner team deprecations and removals process](https://youtu.be/ehT1xBajCRI) <details> <summary> #### Original issue description </summary> We have implemented a JWT version of CI_JOB_TOKEN and tried to roll it out in https://gitlab.com/gitlab-org/gitlab/-/issues/497392+. We had to disable the feature flag because: 1. The new JWT broke an existing implementation of AWS Fargate runner custom executor. This was then [fixed](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/fargate/-/merge_requests/97) but it now requires affected customers to [upgrade to v0.5.1](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/fargate/-/releases/v0.5.1). 2. Other customers also reported that the much longer JWT broke their use of `echo $CI_JOB_TOKEN | base64` pipe command which now prints the value across multiple columns by default. This latter problem can be fixed by using `| base64 -w0` instead. ## Requirements on JWT The switch to use JWT is important for a number of features: * Embed Composite Identity inside CI_JOB_TOKEN * Encoding granular permissions for CI_JOB_TOKEN * https://gitlab.com/gitlab-org/gitlab/-/issues/489855+ * CI now maintains 2 versions of the CI_JOB_TOKEN and we need to remove the maintenance burden. 1. Existing/legacy encrypted string. [This is generated](https://gitlab.com/gitlab-org/gitlab/-/blob/3faf3ee65cf0189030109ac9156c5d81d13f6f7e/app/models/ci/build.rb#L321) when the job goes into `pending` state and persisted in the database. Then, when the job completes we [remove it](https://gitlab.com/gitlab-org/gitlab/-/blob/3faf3ee65cf0189030109ac9156c5d81d13f6f7e/app/workers/ci/build_finished_worker.rb#L43). 2. New JWT implementation. This is generated on the fly when we send the token to the runner. ## Proposal Issue ~"breaking change" warning now and rollout https://gitlab.com/gitlab-org/gitlab/-/issues/497392+ in %"18.0" </details>