feat(datastore): credential table seam and sweep loop
Implements plan step 7 of
docs/plans/2026-08-03-s04b-rotation-tooling.md:
the CredentialTable seam and the credential sweep loop
(internal/datastore/credential_sweep.go), per the spec's
"Credential table seam" section.
The seam is inverted: the loop owns the algorithm and is the only holder
of the crypto.NamespaceKeyReEncryptor — format implementations (#417 (closed))
supply just two queries and never touch key material. The loop drains
each registered table in registration order with constant-size keyset
batches, re-wraps each listed row, writes back under the read-bytes
optimistic guard, counts updated=false as skipped (never retried
in-pass), reports cumulative per-table progress after every batch that
begins processing, and fails closed on defective implementations
(over-long batch, non-advancing cursor, duplicate TableName) with
zero write-backs from a defective batch. On abort, the failing table's
partial counts are kept in its result row; later tables are never
invoked. crypto.ErrStaleActiveKey propagates errors.Is-matchable.
Step 8 composes this loop into the re-encrypt --scope=namespace sweep
phase and owns the registry, exit codes, and log lines. The
CredentialTable doc comment carries the implementer obligations the
loop depends on (single-statement guarded write, caller-owned
WrappedDEK buffers, bytewise-ascending id order, limit bound).
Size
1,333 reviewable LOC against the ~500 guideline: the implementation is
304 lines; the rest is the fake-table behavior matrix the plan bundles
into this step by design ("the fake-table suite carries the loop's full
behavior matrix"). The suite runs a real re-encryptor stack (real
NamespaceKeyReEncryptor, real DefaultNamespaceKeyManager, real
LocalProvider) over in-memory fakes, so the loop's semantics are
proven without a database.
The plan estimated ~200 LOC of implementation and ~400 of tests for
this step (docs/plans/2026-08-03-s04b-rotation-tooling.md:218);
delivered is roughly 1.5x on implementation and 2.6x on tests. The
gap is mutation-driven: each review cycle's coverage findings added
targeted subtests rather than restructuring the suite, so the ratio
of behavior pinned to LOC stayed high even as the file grew past the
estimate.
E2E scenario catalogs
No scenario added or affected: the catalogs cover format-client journeys; this MR is an internal seam with no user-facing surface until Step 8 activates it.
Database review
No new migration, table, or query-producing method — the loop consumes the seam interface; the SQL lands with #417 (closed)'s per-format tables.
Spec coverage
Spec: docs/specs/S04-b-encryption-key-rotation-tooling.md Scope: the AC 6 loop clauses, AC 7's loop mechanism, and the sweep-seam Error Cases rows (this step's ownership per the plan). Command-level clauses (exit codes, log lines, TTL/shred/no-key guards) are Step 8's.
Acceptance criteria
| # | Criterion (clause) | Tests |
|---|---|---|
| AC-6a | Sweep invokes each registered table once, in registration order | TestSweepCredentialTables_RunsTablesInRegistrationOrder |
| AC-6b | Active version passed as the selection bound (namespace, uuid.Nil cursor, constant limit within 500-1000) | TestSweepCredentialTables_RunsTablesInRegistrationOrder |
| AC-6c | Each returned row re-wrapped through ReEncryptRow | TestSweepCredentialTables_WriteBack |
| AC-6d | Write-back via UpdateRowKey under the read-bytes guard, id+version from the re-encryptor | TestSweepCredentialTables_WriteBack (incl. the drifted-version metadata heal) |
| AC-6e | updated=false counts as skipped, never retried in-pass, never clobbered | TestSweepCredentialTables_UpdatedFalseCountsAsSkipped |
| AC-6f | Each table's updated and skipped counts reported (per-batch cumulative, per-table totals) | TestSweepCredentialTables_RunsTablesInRegistrationOrder, TestSweepCredentialTables_ProgressIsCumulativePerBatch, TestSweepCredentialTables_ZeroRowTableReportsZeroTotals, TestSweepCredentialTables_ListErrorAfterFirstBatchKeepsCounts (the log lines themselves: Step 8) |
| AC-6g | First table error aborts the sweep without invoking later tables | TestSweepCredentialTables_TableErrorAbortsSweep |
| AC-6h | Zero registered tables complete successfully | TestSweepCredentialTables_ZeroTables (exit 0 and the log line: Step 8) |
| AC-6i | Shredded namespace / no-key-rows refusals | Step 8 (command guard before the loop). Not this MR. |
| AC-6j | Over-long batch or non-advancing cursor aborts as a defective implementation | TestSweepCredentialTables_BatchOverLimitAborts, TestSweepCredentialTables_CursorNotAdvancingAborts |
| AC-6k | Duplicate TableName rejected before any table runs | TestSweepCredentialTables_DuplicateTableNameRejectedBeforeAnyTableRuns (exit 1: Step 8) |
| AC-7 | A row failing the stale-active guard aborts the sweep (loop mechanism) | TestSweepCredentialTables_StaleActiveKeyAborts (exit 1: Step 8) |
| AC-1..5, 8..13 | Root scope, rotate phase, retirement, CLI validation, configuration | Owned by other plan steps (1, 3-6, 8). Not this MR. |
Error cases
| # | Condition | Tests |
|---|---|---|
| E-sweep-1 | Sweep encounters a row newer than the resolved active key: ErrStaleActiveKey | TestSweepCredentialTables_StaleActiveKeyAborts |
| E-sweep-2 | Sweep with zero registered credential tables: success | TestSweepCredentialTables_ZeroTables |
| E-sweep-3 | A table's read or write fails mid-run: abort, partial count, later tables untouched | TestSweepCredentialTables_TableErrorAbortsSweep, TestSweepCredentialTables_ListErrorAfterFirstBatchKeepsCounts |
| E-sweep-4 | Two registered tables share a TableName: refused before any table runs | TestSweepCredentialTables_DuplicateTableNameRejectedBeforeAnyTableRuns |
| E-sweep-5 | A table returns more rows than requested, or a non-advancing cursor: abort | TestSweepCredentialTables_BatchOverLimitAborts, TestSweepCredentialTables_CursorNotAdvancingAborts |
| Other rows | CLI, root scope, rotate/sweep guards, retirement, DB reachability | Owned by other plan steps. Not this MR. |
Security considerations
| # | Concern | Tests |
|---|---|---|
| S-1 | Security reviews gate delivery | Process control, not testable in code. |
| S-2 | Key material confined to internal/crypto; the sweep transports key-wrap metadata only | Structural: CredentialKeyRow carries no ciphertext and the loop alone holds the re-encryptor; exercised by every loop test. |
| S-3 | No log output carries key material | The progress callback carries (table, updated, skipped) only, pinned by every progress assertion; log lines are Step 8's. |
| S-4..7 | Access control, retention window, fleet configuration, FIPS | Deployment/runbook/service-wide concerns owned outside this step. Not this MR. |
Resolved spec ambiguities (readings pinned by tests)
- Tripwire timing: an over-long or non-advancing batch aborts on receipt; no row of the defective batch is written back.
- Results on abort: the returned
[]TableSweepResultcarries completed tables' totals plus the failing table's partial row; never-invoked tables get no row. - Progress cadence: cumulative per table, after each batch that begins processing — a tripwire-rejected batch reports nothing, an aborting batch reports its partials before the error propagates, and a zero-row pass reports nothing while still returning a zero-total result row.
- Guard-rail sentinels (
errDuplicateTableName,errBatchOverLimit,errCursorNotAdvancing) are unexported, mirroringinternal/crypto/rotation.go's guard rails: the spec names no error identities for them and no caller needserrors.Ismatching. - A sweep whose registry is empty may pass a nil re-encryptor; it is dereferenced only per listed row (Step 8's zero-registration path).
Related to https://gitlab.com/gitlab-org/ops/artifact-registry/-/issues/496