feat(datastore): credential table seam and sweep loop

Implements plan step 7 of docs/plans/2026-08-03-s04b-rotation-tooling.md: the CredentialTable seam and the credential sweep loop (internal/datastore/credential_sweep.go), per the spec's "Credential table seam" section.

The seam is inverted: the loop owns the algorithm and is the only holder of the crypto.NamespaceKeyReEncryptor — format implementations (#417 (closed)) supply just two queries and never touch key material. The loop drains each registered table in registration order with constant-size keyset batches, re-wraps each listed row, writes back under the read-bytes optimistic guard, counts updated=false as skipped (never retried in-pass), reports cumulative per-table progress after every batch that begins processing, and fails closed on defective implementations (over-long batch, non-advancing cursor, duplicate TableName) with zero write-backs from a defective batch. On abort, the failing table's partial counts are kept in its result row; later tables are never invoked. crypto.ErrStaleActiveKey propagates errors.Is-matchable.

Step 8 composes this loop into the re-encrypt --scope=namespace sweep phase and owns the registry, exit codes, and log lines. The CredentialTable doc comment carries the implementer obligations the loop depends on (single-statement guarded write, caller-owned WrappedDEK buffers, bytewise-ascending id order, limit bound).

Size

1,333 reviewable LOC against the ~500 guideline: the implementation is 304 lines; the rest is the fake-table behavior matrix the plan bundles into this step by design ("the fake-table suite carries the loop's full behavior matrix"). The suite runs a real re-encryptor stack (real NamespaceKeyReEncryptor, real DefaultNamespaceKeyManager, real LocalProvider) over in-memory fakes, so the loop's semantics are proven without a database.

The plan estimated ~200 LOC of implementation and ~400 of tests for this step (docs/plans/2026-08-03-s04b-rotation-tooling.md:218); delivered is roughly 1.5x on implementation and 2.6x on tests. The gap is mutation-driven: each review cycle's coverage findings added targeted subtests rather than restructuring the suite, so the ratio of behavior pinned to LOC stayed high even as the file grew past the estimate.

E2E scenario catalogs

No scenario added or affected: the catalogs cover format-client journeys; this MR is an internal seam with no user-facing surface until Step 8 activates it.

Database review

No new migration, table, or query-producing method — the loop consumes the seam interface; the SQL lands with #417 (closed)'s per-format tables.

Spec coverage

Spec: docs/specs/S04-b-encryption-key-rotation-tooling.md Scope: the AC 6 loop clauses, AC 7's loop mechanism, and the sweep-seam Error Cases rows (this step's ownership per the plan). Command-level clauses (exit codes, log lines, TTL/shred/no-key guards) are Step 8's.

Acceptance criteria

# Criterion (clause) Tests
AC-6a Sweep invokes each registered table once, in registration order TestSweepCredentialTables_RunsTablesInRegistrationOrder
AC-6b Active version passed as the selection bound (namespace, uuid.Nil cursor, constant limit within 500-1000) TestSweepCredentialTables_RunsTablesInRegistrationOrder
AC-6c Each returned row re-wrapped through ReEncryptRow TestSweepCredentialTables_WriteBack
AC-6d Write-back via UpdateRowKey under the read-bytes guard, id+version from the re-encryptor TestSweepCredentialTables_WriteBack (incl. the drifted-version metadata heal)
AC-6e updated=false counts as skipped, never retried in-pass, never clobbered TestSweepCredentialTables_UpdatedFalseCountsAsSkipped
AC-6f Each table's updated and skipped counts reported (per-batch cumulative, per-table totals) TestSweepCredentialTables_RunsTablesInRegistrationOrder, TestSweepCredentialTables_ProgressIsCumulativePerBatch, TestSweepCredentialTables_ZeroRowTableReportsZeroTotals, TestSweepCredentialTables_ListErrorAfterFirstBatchKeepsCounts (the log lines themselves: Step 8)
AC-6g First table error aborts the sweep without invoking later tables TestSweepCredentialTables_TableErrorAbortsSweep
AC-6h Zero registered tables complete successfully TestSweepCredentialTables_ZeroTables (exit 0 and the log line: Step 8)
AC-6i Shredded namespace / no-key-rows refusals Step 8 (command guard before the loop). Not this MR.
AC-6j Over-long batch or non-advancing cursor aborts as a defective implementation TestSweepCredentialTables_BatchOverLimitAborts, TestSweepCredentialTables_CursorNotAdvancingAborts
AC-6k Duplicate TableName rejected before any table runs TestSweepCredentialTables_DuplicateTableNameRejectedBeforeAnyTableRuns (exit 1: Step 8)
AC-7 A row failing the stale-active guard aborts the sweep (loop mechanism) TestSweepCredentialTables_StaleActiveKeyAborts (exit 1: Step 8)
AC-1..5, 8..13 Root scope, rotate phase, retirement, CLI validation, configuration Owned by other plan steps (1, 3-6, 8). Not this MR.

Error cases

# Condition Tests
E-sweep-1 Sweep encounters a row newer than the resolved active key: ErrStaleActiveKey TestSweepCredentialTables_StaleActiveKeyAborts
E-sweep-2 Sweep with zero registered credential tables: success TestSweepCredentialTables_ZeroTables
E-sweep-3 A table's read or write fails mid-run: abort, partial count, later tables untouched TestSweepCredentialTables_TableErrorAbortsSweep, TestSweepCredentialTables_ListErrorAfterFirstBatchKeepsCounts
E-sweep-4 Two registered tables share a TableName: refused before any table runs TestSweepCredentialTables_DuplicateTableNameRejectedBeforeAnyTableRuns
E-sweep-5 A table returns more rows than requested, or a non-advancing cursor: abort TestSweepCredentialTables_BatchOverLimitAborts, TestSweepCredentialTables_CursorNotAdvancingAborts
Other rows CLI, root scope, rotate/sweep guards, retirement, DB reachability Owned by other plan steps. Not this MR.

Security considerations

# Concern Tests
S-1 Security reviews gate delivery Process control, not testable in code.
S-2 Key material confined to internal/crypto; the sweep transports key-wrap metadata only Structural: CredentialKeyRow carries no ciphertext and the loop alone holds the re-encryptor; exercised by every loop test.
S-3 No log output carries key material The progress callback carries (table, updated, skipped) only, pinned by every progress assertion; log lines are Step 8's.
S-4..7 Access control, retention window, fleet configuration, FIPS Deployment/runbook/service-wide concerns owned outside this step. Not this MR.

Resolved spec ambiguities (readings pinned by tests)

  • Tripwire timing: an over-long or non-advancing batch aborts on receipt; no row of the defective batch is written back.
  • Results on abort: the returned []TableSweepResult carries completed tables' totals plus the failing table's partial row; never-invoked tables get no row.
  • Progress cadence: cumulative per table, after each batch that begins processing — a tripwire-rejected batch reports nothing, an aborting batch reports its partials before the error propagates, and a zero-row pass reports nothing while still returning a zero-total result row.
  • Guard-rail sentinels (errDuplicateTableName, errBatchOverLimit, errCursorNotAdvancing) are unexported, mirroring internal/crypto/rotation.go's guard rails: the spec names no error identities for them and no caller needs errors.Is matching.
  • A sweep whose registry is empty may pass a nil re-encryptor; it is dereferenced only per listed row (Step 8's zero-registration path).

Related to https://gitlab.com/gitlab-org/ops/artifact-registry/-/issues/496

Edited by Suleimi Ahmed

Merge request reports

Loading
Loading