JWT CI/CD job tokens rejected after the instance token prefix changes
Problem
Running jobs that use a JWT CI/CD job token stop authenticating when the instance token prefix changes or when custom_prefix_for_all_token_types is disabled. The runner gets 403 on PUT /api/v4/jobs/:id and on
trace updates, and the job stays running until it times out. API calls that use CI_JOB_TOKEN from inside the job fail too.
Disabling the feature flag is a likely rollback step while the feature is behind it. Doing that would cause this failure for every running job that uses a JWT job token. This should be fixed before
custom_prefix_for_all_token_types is enabled by default.
Found during review of !258680 (comment 4099427912).
Steps to reproduce
- Enable
jwt_ci_cd_job_token_enabledon a top-level group, enablecustom_prefix_for_all_token_types, and set the instance token prefix toacme. - Start a job that runs
sleep 150and thencurl -H "JOB-TOKEN: $CI_JOB_TOKEN" "$CI_API_V4_URL/job". - While the job sleeps, change the prefix to
beta, or disable the feature flag. - The runner log shows
403onPUT /api/v4/jobs/:idand on the trace update, and the job staysrunning.
Cause
A token keeps the prefix it was created with, but two checks compare it against only the bare glcbt- prefix and the current instance prefix:
Ci::JobToken::Jwt.decode(lib/ci/job_token/jwt.rb)Authn::Tokens::CiJobToken.prefix?(lib/authn/tokens/ci_job_token.rb), which guards the database-token fallback inCi::AuthJobFinderand the job-token handling inGitlab::Auth::AuthFinders
An acme-glcbt-… token fails both checks once the current prefix is beta or empty.
Not affected:
- Setting a prefix for the first time or enabling the flag, because bare
glcbt-tokens are always accepted - Database-backed job tokens, which never carry the instance prefix
Proposal
Match the instance prefix by its format instead of its current value, as Ci::Runner and Ci::Trigger already do with Authn::TokenField::PrefixHelper.optional_instance_prefix_pattern. Use the same pattern in both
checks.
The JWT signature check still decides whether a token is accepted. The prefix only tells the code where the JWT starts, so accepting any well-formed instance prefix doesn't weaken authentication.
Authn::Tokens::StatelessAccessToken#matched_prefix (ee/lib/authn/tokens/stateless_access_token.rb) uses the same "bare or current prefix" logic and should get the same fix.
Acceptance criteria
- A JWT job token created under one instance prefix still authenticates after the prefix changes, is cleared, or
custom_prefix_for_all_token_typesis disabled. -
Authn::Tokens::CiJobToken.prefix?recognizes job tokens with any well-formed instance prefix. -
StatelessAccessTokenhandles prefix changes the same way. - Specs cover the prefix-change and flag-disable cases.