Rehash token digests found under an older db_key_base salt

Rémy Coutablerequested to merge
631700-db-key-base-digest-rehash into master

What does this MR do and why?

Rewrites a token digest with the current db_key_base salt when the token is used and the digest was found under an older salt ("rehash on auth"). With a single key, it's a no-op.

Only the digest is stored, so a new digest can only be computed when the plaintext token is presented. The Phase 4 re-encryption can't do it. Without the rehash, the old salt could only be retired once every pre-rotation token has expired or been revoked. Runner controller tokens never expire.

In Authn::TokenField::Digest#find_token_authenticatable, the digest is rewritten when it doesn't match the current salt. It follows PersonalAccessTokens::LastUsedService:

  • Allowed on GET (PreventWritesOnGet.allow_write_on_get).
  • No sticky writes (LoadBalancing::SessionMap#without_sticky_writes).
  • Skipped on read-only databases (Geo secondaries), frozen records, uniqueness checks and inside an open transaction.
  • Throttled by an exclusive lease per token (10 minutes, not released), taken before the write. A failing write backs off too.
  • Conditional: the UPDATE only matches the digest that was read, so a stale read can't revive a token reset since.
  • Each rehash is logged to auth.log (record type, id and field, never the digest), so the remaining old-salt tokens can be watched.
  • ActiveRecord::ActiveRecordError and Redis::BaseError (from the lease) are tracked with Gitlab::ErrorTracking under their class name only, since the message can carry the digest. They never fail the lookup.

Revoked and expired tokens are rehashed too, because the lookup runs before validation. They stay revoked or expired, and keep getting their specific error once the old salt is retired.

Rolling deploys and retiring the old salt

Step 1 of the runbook puts [new, old] on every node, including Geo secondaries, before any node makes the new key current. So every node finds digests salted with either key. Step 1 must reach every node: once the primary rehashes, a node without the new key rejects pre-rotation tokens too. For the same reason, the new key can't be removed once any node has made it current.

During step 2, nodes still on [new, old] may rehash a digest back to the old salt. Lookups still work, and the lease caps it at one write per token per 10 minutes instead of one per request.

So a token is only reliably on the new salt after its first use more than 10 minutes after step 2 ends on every node. Tokens used only on Geo secondaries are never rehashed. The status report (!259024) should count coverage from that point.

Why there's no feature flag

With a single key, which is the case on GitLab.com and on every instance until a rotation, this MR only adds three steps to each lookup: one more salted SHA-256 of the token (the same digest the lookup always computed), a read of the digest just loaded, and a string comparison that returns early. The lease, the UPDATE, the logging and the error tracking only run when the stored digest uses an older key, which can't happen without a rotation. The lookup with every key is already on master, from !258947 (merged).

A flag placed after the comparison would never be evaluated on GitLab.com, so it wouldn't derisk anything. To cover the three new steps, it would have to be checked on every token authentication. The spec "does not write when the digest already uses the current key" covers the single-key path, and checks that no lease is taken.

If a flag is still wanted, it would go first in rehash_digest, checked against Feature.current_request for a percentage-of-requests rollout, and leave the lookup itself unflagged.

Database query

The rehash runs one UPDATE per rehashed token, by primary key. ci_runner_controller_tokens runs the same statement.

UPDATE "personal_access_tokens"
SET "token_digest" = $1
WHERE "personal_access_tokens"."id" = $2
  AND "personal_access_tokens"."token_digest" = $3

On a gitlab-production-main Database Lab clone, it's a primary key index scan that updates one row in 2.7 ms (4.2 ms with planning), 88 buffer hits, 2 reads and 1 dirtied. The plan below has the values replaced by placeholders. The clone row's digest was set to the old-digest value first, so the UPDATE matches it.

Plan 
 Update on public.personal_access_tokens  (cost=0.44..3.46 rows=0 width=0) (actual time=2.682..2.682 rows=0 loops=1)
   Buffers: shared hit=88 read=2 dirtied=1
   WAL: records=15 fpi=1 bytes=9010
   I/O Timings: read=2.261 write=0.000
   ->  Index Scan using personal_access_tokens_pkey on public.personal_access_tokens  (cost=0.44..3.46 rows=1 width=38) (actual time=0.073..0.074 rows=1 loops=1)
         Index Cond: (personal_access_tokens.id = $2)
         Filter: ((personal_access_tokens.token_digest)::text = $3::text)
         Buffers: shared hit=6
         WAL: records=1 fpi=0 bytes=56
         I/O Timings: read=0.000 write=0.000
Settings: effective_cache_size = '472585MB', work_mem = '230MB', jit = 'off', random_page_cost = '1.5', seq_page_cost = '4'

The update isn't HOT, since token_digest is indexed: the 15 WAL records include the index entries. With the lease, that's at most one such write per token every 10 minutes during a rotation, and none otherwise.

References

Screenshots or screen recordings

Not applicable, backend only.

How to set up and validate locally

  1. Run the specs:

    bin/rspec spec/lib/authn/token_field/digest_spec.rb \
  spec/requests/api/personal_access_tokens/self_information_spec.rb \
  ee/spec/models/ci/runner_controller_token_spec.rb

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code

Edited by Rémy Coutable

Merge request reports