Trim excess last_used_ips down to five per token
What does this MR do and why?
Adds a batched background migration that trims each personal access token's
personal_access_token_last_used_ips down to its five most recent distinct IPs
(newest occurrence per IP by created_at), cleaning up both the excess rows and
the duplicate-IP rows that accumulated. This matches the write-path dedupe in
!250690 (merged).
The field is documented as "the five most recent unique IP addresses", but a
write-path bug let tokens accumulate more: the trim's ip_count guard read a
replica that lagged the just-appended row, undercounted, and skipped the trim.
About 14% of tokens (583,414) hold more than five IPs as a result. The write
path is fixed in !250690 (merged);
this MR cleans up the backlog that has already accumulated.
Deleting rows beyond the newest five matches the documented behaviour, so it is not a surprising loss of audit data. The job is idempotent and safe to re-run.
Must merge after !250690 (merged), otherwise the still-buggy write path re-accumulates the excess.
Part of #616954 (closed) (step 3).
Database review
The job deletes per sub-batch with a single DELETE. It first collapses each
token's duplicate IPs to their newest row (DISTINCT ON (personal_access_token_id, ip_address)), ranks those distinct IPs per token by recency, keeps the five most
recent, and deletes everything else for the affected tokens:
WITH affected_tokens AS MATERIALIZED (
SELECT DISTINCT personal_access_token_id
FROM (<sub_batch of ids>) batch
),
newest_per_ip AS (
SELECT DISTINCT ON (personal_access_token_id, ip_address)
id, personal_access_token_id, created_at
FROM personal_access_token_last_used_ips
WHERE personal_access_token_id IN (SELECT personal_access_token_id FROM affected_tokens)
ORDER BY personal_access_token_id, ip_address, created_at DESC, id DESC
),
ids_to_keep AS (
SELECT id
FROM (
SELECT
id,
row_number() OVER (
PARTITION BY personal_access_token_id
ORDER BY created_at DESC, id DESC
) AS rn
FROM newest_per_ip
) ranked
WHERE rn <= 5
)
DELETE FROM personal_access_token_last_used_ips
WHERE personal_access_token_id IN (SELECT personal_access_token_id FROM affected_tokens)
AND id NOT IN (SELECT id FROM ids_to_keep);Per-token row counts are small (five in steady state, a low-hundreds tail), and
the batch is cursor-based over personal_access_token_last_used_ips.
EXPLAIN query plan: to be captured on Database Lab or a read replica against a
real batch of over-cap ids (substitute for <sub_batch of ids>), reading the
actual ... rows= values, not the cost=... rows= estimates.
Post-deploy validation
After this migration finalizes, this query (Database Lab or a read replica) should return ~0 rows, i.e. no token holds more than five IPs or any duplicate IP:
SELECT personal_access_token_id
FROM personal_access_token_last_used_ips
GROUP BY personal_access_token_id
HAVING count(*) > 5 OR count(*) > count(DISTINCT ip_address);It returns 583,414 today (over-cap tokens); the count(*) > count(DISTINCT ip_address) clause additionally flags any token still holding duplicate rows for
the same IP.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist.