Trim excess last_used_ips down to five per token

What does this MR do and why?

Adds a batched background migration that trims each personal access token's personal_access_token_last_used_ips down to its five most recent distinct IPs (newest occurrence per IP by created_at), cleaning up both the excess rows and the duplicate-IP rows that accumulated. This matches the write-path dedupe in !250690 (merged).

The field is documented as "the five most recent unique IP addresses", but a write-path bug let tokens accumulate more: the trim's ip_count guard read a replica that lagged the just-appended row, undercounted, and skipped the trim. About 14% of tokens (583,414) hold more than five IPs as a result. The write path is fixed in !250690 (merged); this MR cleans up the backlog that has already accumulated.

Deleting rows beyond the newest five matches the documented behaviour, so it is not a surprising loss of audit data. The job is idempotent and safe to re-run.

Must merge after !250690 (merged), otherwise the still-buggy write path re-accumulates the excess.

Part of #616954 (closed) (step 3).

Database review

The job deletes per sub-batch with a single DELETE. It first collapses each token's duplicate IPs to their newest row (DISTINCT ON (personal_access_token_id, ip_address)), ranks those distinct IPs per token by recency, keeps the five most recent, and deletes everything else for the affected tokens:

WITH affected_tokens AS MATERIALIZED (
  SELECT DISTINCT personal_access_token_id
  FROM (<sub_batch of ids>) batch
),
newest_per_ip AS (
  SELECT DISTINCT ON (personal_access_token_id, ip_address)
    id, personal_access_token_id, created_at
  FROM personal_access_token_last_used_ips
  WHERE personal_access_token_id IN (SELECT personal_access_token_id FROM affected_tokens)
  ORDER BY personal_access_token_id, ip_address, created_at DESC, id DESC
),
ids_to_keep AS (
  SELECT id
  FROM (
    SELECT
      id,
      row_number() OVER (
        PARTITION BY personal_access_token_id
        ORDER BY created_at DESC, id DESC
      ) AS rn
    FROM newest_per_ip
  ) ranked
  WHERE rn <= 5
)
DELETE FROM personal_access_token_last_used_ips
WHERE personal_access_token_id IN (SELECT personal_access_token_id FROM affected_tokens)
  AND id NOT IN (SELECT id FROM ids_to_keep);

Per-token row counts are small (five in steady state, a low-hundreds tail), and the batch is cursor-based over personal_access_token_last_used_ips.

EXPLAIN query plan: to be captured on Database Lab or a read replica against a real batch of over-cap ids (substitute for <sub_batch of ids>), reading the actual ... rows= values, not the cost=... rows= estimates.

Post-deploy validation

After this migration finalizes, this query (Database Lab or a read replica) should return ~0 rows, i.e. no token holds more than five IPs or any duplicate IP:

SELECT personal_access_token_id
FROM personal_access_token_last_used_ips
GROUP BY personal_access_token_id
HAVING count(*) > 5 OR count(*) > count(DISTINCT ip_address);

It returns 583,414 today (over-cap tokens); the count(*) > count(DISTINCT ip_address) clause additionally flags any token still holding duplicate rows for the same IP.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Eduardo Sanz García

Merge request reports

Loading
Loading