Report when older db_key_base salts can be retired

Rémy Coutablerequested to merge
631458-db-key-base-token-digests-status into master

What does this MR do and why?

Adds Gitlab::Encryption::DbKeyBaseRotation::TokenDigestsStatus. It reports when the old db_key_base salt can be retired for token digests. EE adds runner controller tokens.

Token digests don't tell which key salted them. Rehashing on use (!259034) rewrites a digest found under an older salt when the token is used. So only active tokens created before the rotation and not used since may still need an older salt.

  • .new(rotated_at:) raises ArgumentError unless rotated_at is a time in the past. It must be when the last node switched to the new key, plus the rehash lease timeout.
  • #execute returns one result per digest token model with remaining, expired, without_expiry and latest_expiry. It iterates each table in batches of 10,000 over the primary key, with one aggregate query per batch. The result is memoized.
  • #retirement_date returns today when none remain, the day after the latest expiry date when all remaining tokens expire, or nil while a remaining token never expires. Runner controller tokens have no expiry, so only use, revocation or rotation clears them.

Nothing calls it yet. The gitlab:db_key_base:status rake task (https://gitlab.com/gitlab-org/gitlab/-/work_items/634003) will, for runbook step 5. It doesn't depend on that code, but is only meaningful once the rehash ships. That task should refuse to report while the rehash isn't enabled, and warn about expired tokens.

When the report can over-count or under-count
  • Over-counts (safe): last_used_at updates are throttled (10 minutes for personal access tokens), so a recently used token may still be counted. Tokens created or used exactly at rotated_at are counted.
  • rotated_at too early (under-counts): a node still on the old key creates old-salt tokens after it, and can rehash tokens back to the old key. Hence "last node switched, plus the rehash lease timeout".
  • Failed rehash (accepted, under-counts): a token whose rehash failed on its last use since the rotation is missed, because last_used_at still moves. A later use retries the rehash once its 10-minute lease expires. The remaining risk is a token used once since the rotation, with that one rehash failing.
  • Expired tokens: they are reported as expired but don't affect retirement_date. Extending their expiry after retirement (for example with gitlab:tokens:edit) would break them.
  • Day after the latest expiry: a personal access token stops working at 00:00 UTC on its expires_at date. The extra day keeps instances ahead of UTC from retiring a few hours early.
What retiring the salt doesn't cover
  • Replaying a revoked (rotated) personal access token revokes its token family. After retirement, a revoked token from before the rotation can't be found, so it gets a plain 401 instead.
  • db_key_base also salts Users::CreditCardValidation hashes and web_hook_logs.url_hash. This report covers token digests only.

Database

Each table is iterated with each_batch over its primary key, because no index serves these filters. The aggregate for each batch:

SELECT COUNT(*) FILTER (WHERE expires_at IS NULL OR expires_at > NOW()),
       COUNT(*) FILTER (WHERE expires_at <= NOW()),
       COUNT(*) FILTER (WHERE expires_at IS NULL),
       MAX(expires_at)
FROM "personal_access_tokens"
WHERE "personal_access_tokens"."id" >= 1 AND "personal_access_tokens"."id" < 10001
  AND ("personal_access_tokens"."revoked" = FALSE OR "personal_access_tokens"."revoked" IS NULL)
  AND "personal_access_tokens"."created_at" <= '2026-10-01 00:00:00'
  AND ("personal_access_tokens"."last_used_at" IS NULL OR "personal_access_tokens"."last_used_at" <= '2026-10-01 00:00:00')
LIMIT 1

Query plan: TODO

EE, on the CI database:

SELECT COUNT(*)
FROM "ci_runner_controller_tokens"
WHERE "ci_runner_controller_tokens"."id" >= 1 AND "ci_runner_controller_tokens"."id" < 10001
  AND "ci_runner_controller_tokens"."status" = 0
  AND "ci_runner_controller_tokens"."created_at" <= '2026-10-01 00:00:00'
  AND ("ci_runner_controller_tokens"."last_used_at" IS NULL OR "ci_runner_controller_tokens"."last_used_at" <= '2026-10-01 00:00:00')

Query plan: TODO

References

Screenshots or screen recordings

Not applicable, backend only.

How to set up and validate locally

  1. Run the specs:

    bin/rspec spec/lib/gitlab/encryption/db_key_base_rotation ee/spec/lib/ee/gitlab/encryption/db_key_base_rotation

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code

Edited by Rémy Coutable

Merge request reports