Add a prefix to CI Build tokens behind a feature flag

What does this MR do and why?

Add a prefix to CI Build tokens behind a feature flag

Prefixes CI Build tokens (a.k.a. CI_JOB_TOKEN) with glcbt- following the guidance at https://docs.gitlab.com/ee/development/secure_coding_guidelines.html#token-prefixes.

GitLab applies a prefix to some of its generated secrets. For example, a Personal Access Token begins with glpat-. This MR adds a prefix to Build Tokens. It also updates our frontend secret detection which helps prevent users from leaking tokens via Issue / MR comments.

Build tokens belong to build jobs and are used to authenticate against the APIs described at https://docs.gitlab.com/ee/ci/jobs/ci_job_token.html

Build tokens were already prefixed with a hexadecimal partition ID. The new static prefix is placed before the existing prefix.

A feature flag is being used to reduce the risk of breaking CI pipelines and/or third-party integrations, which might have made assumptions about the format of GitLab's build tokens remaining static. The flag can be enabled or disabled per namespace project.

Resolves Add prefix to CI Job Tokens (#426137 - closed). The issue includes discussion of the prefix and risks of change.

Changelog: changed

Rollout issue: [Feature flag] Rollout of `prefix_ci_build_tokens` (gitlab-com/gl-infra/production#17299 - closed)

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

I have followed the checklist. I believe the risks are mitigated by rolling out with a group-based FF.

Screenshots or screen recordings

This change is invisible to the user - CI_JOB_TOKEN should never be seen, only used behind the scenes when jobs are run.

How to set up and validate locally

Note: the tokens in these steps are all A) already expired because their jobs finished and B) for the local GDK development environment. There is no security risk including them, and it assists with understanding the change.

Validating the current state:

1. Ensure a runner is configured and running locally: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/runner.md
2. Create a project
3. Click add file, use the file name .gitlab-ci.yml, and the content below

   # To validate build token prefix.
   # Don't run on .com or you'll leak a sensitive value.
   before_script:
       - apk add --no-cache curl
   insecure:
     stage: build
     script:
       - echo "Attempting to echo CI_JOB_TOKEN."
       - echo $CI_JOB_TOKEN
       - echo "Attempting to echo base64(CI_JOB_TOKEN)"
       - echo $CI_JOB_TOKEN | base64
       - 'curl --location --insecure --output artifacts.zip "https://gdk.test:3443/api/v4/projects/$CI_PROJECT_ID/jobs/artifacts/main/download?job=insecure&job_token=$CI_JOB_TOKEN"'

4. Commit to main/master
5. View the job

   $ echo "Attempting to echo CI_JOB_TOKEN."
   Attempting to echo CI_JOB_TOKEN.
   $ echo $CI_JOB_TOKEN
   [MASKED]
   $ echo "Attempting to echo base64(CI_JOB_TOKEN)"
   Attempting to echo base64(CI_JOB_TOKEN)
   $ echo $CI_JOB_TOKEN | base64
   NjRfM1JMM0dBOGFzQmFLNFlvbnp2eWcK

   1. Note that the raw CI_JOB_TOKEN is masked in output (great!)
   2. Note that we can avoid the masking by base64 encoding it first (insecure!)
   3. Note that you can decode the base64 version and get the plaintext token (expected)

      % echo "NjRfM1JMM0dBOGFzQmFLNFlvbnp2eWcK" | base64 -d
      64_3RL3GA8asBaK4Yonzvyg

   4. Note that cURL executes successfully; CI_JOB_TOKEN is valid authentication/authorization

Validating the change:

1. Check out 426137-prefix-ci-tokens and gdk restart.
2. Enable the feature flag for a group that isn't a parent of the project, e.g. Feature.enable(:prefix_ci_build_tokens, Group.first)
   1. Re-run the job
   2. Observe that the job token echoed out still lacks a prefix

      ...
      $ echo $CI_JOB_TOKEN | base64
      NjRfb29lbnd4b01MZFRBcmtfR3Jqak4K

      % echo "NjRfb29lbnd4b01MZFRBcmtfR3Jqak4K" | base64 -d
      64_ooenwxoMLdTArk_GrjjN

3. Enable the feature flag for this project's group, e.g. Feature.enable(:prefix_ci_build_tokens, Group.find(GROUP_ID))
   1. Re-run the job

      $ echo "Attempting to echo CI_JOB_TOKEN."
      Attempting to echo CI_JOB_TOKEN.
      $ echo $CI_JOB_TOKEN
      [MASKED]
      $ echo "Attempting to echo base64(CI_JOB_TOKEN)"
      Attempting to echo base64(CI_JOB_TOKEN)
      $ echo $CI_JOB_TOKEN | base64
      Z2xjYnQtNjRfVHRqejZTY3E2ck1Mc0I3SGY1MUIK

      % echo "Z2xjYnQtNjRfVHRqejZTY3E2ck1Mc0I3SGY1MUIK" | base64 -d
      glcbt-64_Ttjz6Scq6rMLsB7Hf51B

      1. Observe that the job token is still correctly masked
      2. Observe that when you base64 decode the value, it has the new prefix
      3. Observe that cURL still executes successfully; prefixed CI_JOB_TOKEN remains valid authentication/authorization /assign me

Related to #426137 (closed)

changed milestone to %16.9

added CI job token Category:Secrets Management devopsverify featureaddition grouppipeline security sectionci security teamProduct Security Engineering typefeature workflowin dev labels

assigned to @nmalcolm

added backend documentation feature flag feature flagexists frontend labels

	1 Message
	This merge request adds or changes documentation files. A review from the Technical Writing team before you merge is recommended. Reviews can happen after you merge.

Documentation review

The following files require a review from a technical writer:

• doc/security/token_overview.md (Link to current live version)

The review does not need to block merging this merge request. See the:

• Metadata for the *.md files that you've changed. The first few lines of each *.md file identify the stage and group most closely associated with your docs change.
• The Technical Writer assigned for that stage and group.
• Documentation workflows for information on when to assign a merge request for review.

Reviewer roulette

Changes that require review have been detected!

Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:

Category	Reviewer	Maintainer
backend	@akotte (UTC+5.5, 7.5 hours behind author)	@tachyons-gitlab (UTC+5.5, 7.5 hours behind author)
frontend	@ramistry (UTC+5.5, 7.5 hours behind author)	@deepika.guliani (UTC+5.5, 7.5 hours behind author)
~"Verify"	Reviewer review is optional for ~"Verify"	@grzesiek (UTC+1, 12 hours behind author)

Please check reviewer's status!

• Reviewer is available!
• Reviewer is unavailable!

Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.

To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.

Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.

If needed, you can retry the danger-review job that generated this comment.

Generated by Danger

mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@66fc256e

Bundle size analysis [beta]

This compares changes in bundle size for entry points between the commits 4afb8e6b and 7974e181

Special assets

Entrypoint / Name	Size before	Size after	Diff	Diff in percent
average	4.12 MB	4.12 MB	-	0.0 %
mainChunk	3.09 MB	3.09 MB	-	0.0 %

---

Note: We do not have exact data for 4afb8e6b. So we have used data from: 47202c36.
The intended commit has no webpack pipeline, so we chose the last commit with one before it.

Please look at the full report for more details

---

Read more about how this report works.

Generated by Danger

E2E Test Result Summary

allure-report-publisher generated test report!

e2e-test-on-gdk: test report for 7974e181

expand test summary

+------------------------------------------------------------------+
|                          suites summary                          |
+-------------+--------+--------+---------+-------+-------+--------+
|             | passed | failed | skipped | flaky | total | result |
+-------------+--------+--------+---------+-------+-------+--------+
| Data Stores | 23     | 0      | 0       | 0     | 23    | ✅     |
| Govern      | 72     | 0      | 0       | 0     | 72    | ✅     |
| Create      | 54     | 0      | 7       | 0     | 61    | ✅     |
| Plan        | 55     | 0      | 0       | 0     | 55    | ✅     |
| Release     | 5      | 0      | 0       | 0     | 5     | ✅     |
| Package     | 15     | 0      | 1       | 0     | 16    | ✅     |
| Monitor     | 7      | 0      | 0       | 0     | 7     | ✅     |
| Verify      | 31     | 0      | 0       | 0     | 31    | ✅     |
| Manage      | 0      | 0      | 1       | 0     | 1     | ➖     |
| Analytics   | 2      | 0      | 0       | 0     | 2     | ✅     |
+-------------+--------+--------+---------+-------+-------+--------+
| Total       | 264    | 0      | 9       | 0     | 273   | ✅     |
+-------------+--------+--------+---------+-------+-------+--------+

e2e-package-and-test: test report for 7974e181

expand test summary

+------------------------------------------------------------------+
|                          suites summary                          |
+-------------+--------+--------+---------+-------+-------+--------+
|             | passed | failed | skipped | flaky | total | result |
+-------------+--------+--------+---------+-------+-------+--------+
| Verify      | 138    | 0      | 27      | 6     | 165   | ✅     |
| Create      | 149    | 0      | 16      | 2     | 165   | ✅     |
| Data Stores | 4      | 0      | 0       | 0     | 4     | ✅     |
| Package     | 0      | 0      | 2       | 0     | 2     | ➖     |
| Plan        | 8      | 0      | 0       | 0     | 8     | ✅     |
| Monitor     | 8      | 0      | 0       | 0     | 8     | ✅     |
| Govern      | 6      | 0      | 0       | 0     | 6     | ✅     |
+-------------+--------+--------+---------+-------+-------+--------+
| Total       | 313    | 0      | 45      | 8     | 358   | ✅     |
+-------------+--------+--------+---------+-------+-------+--------+

Two errors are

https://gitlab.com/gitlab-org/gitlab/-/jobs/5780816124

# [RSpecRunTime] RSpec elapsed time: 81 minutes 34.38 seconds. Current RSS: ~1286M
ERROR: Job failed: execution took longer than 1h30m0s seconds

https://gitlab.com/gitlab-org/gitlab/-/jobs/5780816267

FATAL: write /builds/gitlab-org/gitlab/public/assets/webpack/sourcegraph/0.0.91/extensions/sourcegraph-graphql/extension.js: no space left on device 

Neither of which seem relevant. Will try re-running those jobs

added 276 commits

• 7e6c7607...1ae21339 - 275 commits from branch master
• 36ce66a8 - Add a prefix to CI Build tokens behind a feature flag

Compare with previous version

added 1 commit

• d537a41b - Add a prefix to CI Build tokens behind a feature flag

Compare with previous version

changed the description

@greg FYI as SC for grouppipeline security, this MR "contains changes to processing or storing of credentials or tokens, authorization, and authentication methods". (We're adding a prefix).

marked this merge request as ready

resolved all threads

requested review from @alexbuijs and @thutterer