Draft: Add bulk RPCs for OAuth applications and refresh tokens

Rails can send OAuth applications and Doorkeeper refresh tokens to IAM in batches of up to 1,000, and ask IAM whether one refresh token is still active. Today the Layer 4 sweep makes one call per row, and the refresh-token backfill covers about 46M rows, so one pass takes days and holds up the refresh routing switch.

The code adds BulkUpsertClients on the clients service, and BulkUpsert, BulkDelete and GetState on the refresh-token service. A bulk upsert returns one UpsertResult per record in input order, so one bad row no longer blocks the others. The shared per-record logic lives in auth/oauth/server/internal/batch, with one counter, iam_auth_batch_records_total, and one log line per call.

This branch is the integration MR and is not merged. The stack below carries the same change in twelve parts; the first four have merged.

Departures from the single-record paths, each on purpose: zero cell and source ids are stored as NULL through NULLIF to match the single path; the bulk SQL uses one unnest per column because sqlc rejects a sixteen-argument unnest; the counter pre-creates eight series, since the clients path can never produce the two client reasons; and server.InitBatchMetrics forwards to the internal package because auth cannot import it. The whitespace rule for scope items, which keeps the space-joined encoding lossless, is its own MR, !1012 (merged).

GetState reads through the unique index on (owning_cell_id, source_id) that !947 (merged) added. !1010 (merged) moves the upsert rules from that statement into Go, so the single and bulk paths share one decision.

Verification: unit and integration suites on Postgres and YugabyteDB, including the bufconn test that sends 2,500 tokens in 1000/1000/500 chunks; /-/metrics shows the eight series at zero after startup; the end-to-end test in system-test-suite (branch sk/e2e/bulk-oauth-rpcs, merges after the IAM release) wrote and deleted 3,000 tokens against a local make run-iam in 0.7s. The GDK Rails sweep run waits on the Rails side of gitlab-org/gitlab#635784 (closed), which still says the bound is 10,000.

Issues: gitlab-org/gitlab#635781 (closed), gitlab-org/gitlab#635782 (closed), gitlab-org/gitlab#632077 (closed) and gitlab-org/gitlab#635135 (closed).

Stack:

  1. !988 (merged) feat(proto): add BulkUpsertClients and UpsertResult (merged)
  2. !989 (merged) feat(auth): add BulkUpsertClients with per-record results (merged)
  3. !990 (merged) refactor(auth): move the bulk record helpers to internal/batch (merged)
  4. !1010 (merged) fix(auth): decide refresh-token upserts in Go (merged)
  5. !1029 (merged) fix(auth): gate stored rows in one upsert helper and wait on the lock (merged)
  6. !1030 (merged) feat(auth): add a batch path to the refresh-token repository (merged)
  7. !1031 (merged) feat(proto): add BulkUpsert to the refresh-token service (merged)
  8. !991 (merged) feat(auth): add refresh-token BulkUpsert with per-record results (merged)
  9. !1069 (merged) feat(auth): refuse upserts for deleted refresh-token ids (merged)
  10. !992 (merged) feat(auth): add refresh-token BulkDelete (merged)
  11. !1110 (merged) feat(auth): mark a deleted client's refresh tokens before the cascade
  12. !993 (merged) feat(auth): add refresh-token GetState
  13. !1118 (merged) test(auth): drive the refresh-token bulk RPCs through the server
  14. !994 (merged) feat(auth): count bulk RPC records by outcome
Edited by Shilpa Kundapur

Merge request reports

Loading
Loading