feat(cmd): namespace-scope rotate and sweep phases
Implements plan step 8 of
docs/plans/2026-08-03-s04b-rotation-tooling.md:
both phases of re-encrypt --scope=namespace, replacing the stub the
operator subcommands shipped.
The rotate phase drives the namespace-key manager's rotation entry
point behind the ActiveKeyAge guard: TTL-not-elapsed refuses at Warn
with the age, the TTL, and an RFC3339 retry time ("if a second rotation
is intended, retry after …"); shredded and no-key-row namespaces refuse
without creating anything; success creates exactly one version and
summarizes at Info with the created key_version.
The sweep phase re-checks the guard (refusing inside one TTL with the
retry time), resolves the selection bound from the active key, and
drives datastore.SweepCredentialTables over the composed registry:
per-batch progress and per-table totals at Info (a failing table logs
"table aborted" with its partial counts), abort or stale-active-key
surfaces as exit 1, and the zero-registration sweep completes
secret-read-free — the key manager is constructed only when at least
one credential table is registered. The registry is composed empty in
production; per-format registrations are #417 (closed)'s scope and inherit the
composition's one-minute statement bound automatically
(statementBoundCredentialTable wraps each table where the registry
enters the sweep). The manager's own store calls are bounded the same
way (statementBoundKeyStore).
No log line or error carries key material (pinned over the root key, KEKs, wrapped keys, and every wrapped DEK on both happy paths).
Stacking
Both parents (!1344 (merged) and !1412 (merged)) have merged, so this targets main and
the diff is the step 8 change set.
Size
~2,470 added lines: ~530 implementation, the rest the unit matrix and the end-to-end integration suite the plan assigns to this step, plus the review-cycle hardening (deadline pins for both new statement-bound decorators, abort-summary and secret-decode pins, TTL boundary case) and the round-1 review fixes (goleak pins on both manager paths, the secret-read-free guard refusals, and the run's log-attribute contract).
Scope beyond the plan's step 8
Round 1 grew the step into internal/crypto, which the plan's step-8
entry now records. crypto.DefaultNamespaceKeyManager gained
RotateKeyVersion, a rotation entry point that publishes nothing and
returns only the version the store assigned; RotateKey is unchanged
for callers that use the key they rotate to.
The rotate action needs a rotation to have happened and never uses the
key, and two review findings were properties of the publishing
primitive rather than of the call site: it hands package main a
plaintext copy that package cannot wipe, and its publication step can
refuse after the row is committed, reporting a live new version as a
failed rotation. Both are fixed at the entry point instead of described
at the call site.
E2E scenario catalogs
No scenario added or affected: operator-only rotation tooling; the catalogs cover format-client journeys.
Database review
No new migration or query-producing method — the phases drive the merged store operations and the step 7 seam.
Spec coverage
Spec: docs/specs/S04-b-encryption-key-rotation-tooling.md
Acceptance criteria
| # | Criterion | Tests |
|---|---|---|
| AC-1 | Root scope re-wraps every live row; re-run is zero rows | Step 4 (on this base): TestReEncryptRootScope_RewrapsEveryLiveRow |
| AC-2 | Single-key file still verifies | Step 4: TestReEncryptRootScope_SingleKeyFileVerifies, TestReEncryptRootScope_SingleKeyFileVerifiesOffender |
| AC-3 | Whole-live-set verification names offenders | Step 4: TestReEncryptRootScope_StrandedURIFailsVerification, TestReEncryptRootScope_RewrapsEveryLiveRow |
| AC-4 | Orientation refusal + startup URI logging | Step 4: TestReEncryptRoot_OrientationRefusalNamesBothURIs, TestReEncryptRootScope_OrientationRefusalLeavesRowsUntouched |
| AC-5 | Rotate creates exactly one version; shredded/no-rows/TTL refusals | TestReEncryptNamespaceScope_RotateCreatesExactlyOneVersion, TestReEncryptNamespaceScope_RotateGuardRefusals, TestReEncryptNamespace_GuardRefusalsWarnAndExit1 (rotate rows) |
| AC-6 | Sweep command-level clauses | TTL refusal with retry time: TestReEncryptNamespace_SweepTTLRefusalNamesEarliestRetryTime, TestReEncryptNamespaceScope_SweepGuardOutcomes/before the ttl elapses; ordered run, selection bound, guarded write-back, skip counting, per-table counts: TestReEncryptNamespaceScope_SweepRunsRegisteredTablesInOrder; abort without later tables + duplicate TableName exit 1: TestReEncryptNamespaceScope_SweepAborts; zero registration: TestReEncryptNamespace_SweepZeroRegistrationSucceeds, TestReEncryptNamespaceScope_SweepGuardOutcomes/zero registered tables; shredded/no-rows: TestReEncryptNamespaceScope_SweepGuardOutcomes, TestReEncryptNamespace_GuardRefusalsWarnAndExit1 (sweep rows). Loop-behavior clauses (tripwire mechanics, duplicate rejection before any table): Step 7, internal/datastore/credential_sweep_test.go |
| AC-7 | ErrStaleActiveKey aborts the sweep with exit 1 |
TestReEncryptNamespaceScope_SweepAborts/stale active key |
| AC-8 | RetireKeyVersion contract and sentinel matrix |
Step 1: internal/datastore/namespace_encryption_keys_rotation_integration_test.go |
| AC-9 | FK refusal mapped to ErrKeyStillReferenced |
Step 1: same file, via the testutil FK-table helper |
| AC-10 | Retirement barrier concurrency | Step 6 (test-only MR, not on this base) |
| AC-11 | Usage-error matrix exits 1 before any database access | Step 3: TestReEncryptCommand_UsageErrorsExitBeforeDatabase, retire_test.go |
| AC-12 | Missing encryption block refused; database-only environment | Steps 2-3: TestRunOperator_MissingEncryptionBlockRefusedBeforeDatabase, TestReEncryptCommand_MissingEncryptionBlockRefused |
| AC-13 | Retire pre-flight logging | Step 5 (not on this base) |
| AC-14 | No key material in captured logs | This MR for the namespace flows: TestReEncryptNamespaceScope_RotateCreatesExactlyOneVersion, TestReEncryptNamespaceScope_SweepRunsRegisteredTablesInOrder (both via assertLogsCarryNoKeyMaterial); other flows: Steps 2, 4, 5 |
Error cases
| # | Condition | Tests |
|---|---|---|
| E-1 | Usage errors | Step 3: TestReEncryptCommand_UsageErrorsExitBeforeDatabase, retire_test.go |
| E-2 | No encryption block | Steps 2-3: TestRunOperator_MissingEncryptionBlockRefusedBeforeDatabase, TestReEncryptCommand_MissingEncryptionBlockRefused |
| E-3 | Root-key secret missing or undecodable | Steps 3-4: TestReEncryptRoot_SecretLoadFailureRefused, TestReEncryptRoot_MalformedSecretRefused, TestValidateRootKeySecret |
| E-4 | Root scope: transient failure mid-run | Step 4: TestReEncryptRootScope_TransientFailureResumes |
| E-5 | Root scope: matched row fails to unwrap | Step 4: TestReEncryptRootScope_UnwrapFailureAbortsAndPersists |
| E-6 | Root scope: duplicate retained line skipped | Step 4: TestReEncryptRootScope_DuplicateLineSkipped |
| E-7 | Root scope: orientation mismatch | Step 4: TestReEncryptRoot_OrientationRefusalNamesBothURIs, TestReEncryptRootScope_OrientationRefusalLeavesRowsUntouched |
| E-8 | Interruption mid-run | Step 2: TestRunOperator_SignalCancelsActionContext, TestRunOperator_SIGINTCancelsActionContext; resume: Step 4 (E-4 test); retirement atomicity: Step 1 |
| E-9 | Root scope: verification finds an offender | Step 4: TestReEncryptRootScope_StrandedURIFailsVerification, TestReEncryptRootScope_SingleKeyFileVerifiesOffender |
| E-10 | Rotate on shredded namespace | TestReEncryptNamespaceScope_RotateGuardRefusals/shredded namespace, TestReEncryptNamespace_GuardRefusalsWarnAndExit1/rotate on shredded namespace |
| E-11 | Rotate on namespace with no key rows | TestReEncryptNamespaceScope_RotateGuardRefusals/no key rows, TestReEncryptNamespace_GuardRefusalsWarnAndExit1/rotate on namespace with no key rows |
| E-12 | Rotate within one TTL | TestReEncryptNamespaceScope_RotateGuardRefusals/within one ttl, TestReEncryptNamespaceScope_RotateCreatesExactlyOneVersion (second invocation), TestReEncryptNamespace_GuardRefusalsWarnAndExit1/rotate within one ttl of the active key |
| E-13 | Sweep before one TTL, earliest retry time | TestReEncryptNamespace_SweepTTLRefusalNamesEarliestRetryTime, TestReEncryptNamespaceScope_SweepGuardOutcomes/before the ttl elapses |
| E-14 | Sweep row newer than resolved active key | TestReEncryptNamespaceScope_SweepAborts/stale active key |
| E-15 | Sweep with zero registered tables | TestReEncryptNamespace_SweepZeroRegistrationSucceeds, TestReEncryptNamespaceScope_SweepGuardOutcomes/zero registered tables |
| E-16 | Sweep table read/write fails mid-run | Loop mechanics (abort, partial counts, later tables not attempted): Step 7, internal/datastore/credential_sweep_test.go; command-level error-to-exit-1 surfacing: TestReEncryptNamespaceScope_SweepAborts/stale active key |
| E-17 | Two registered tables share a TableName |
Loop rejection before any table: Step 7; command-level exit 1: TestReEncryptNamespaceScope_SweepAborts/duplicate table name |
| E-18 | Over-long batch or non-advancing cursor | Tripwire mechanics: Step 7, internal/datastore/credential_sweep_test.go; the command surfaces any loop error as exit 1 (path pinned by TestReEncryptNamespaceScope_SweepAborts/stale active key) |
| E-19 | Sweep on shredded namespace | TestReEncryptNamespaceScope_SweepGuardOutcomes/shredded namespace, TestReEncryptNamespace_GuardRefusalsWarnAndExit1/sweep on shredded namespace |
| E-20 | Sweep on namespace with no key rows (no auto-create) | TestReEncryptNamespaceScope_SweepGuardOutcomes/no key rows, TestReEncryptNamespace_GuardRefusalsWarnAndExit1/sweep on namespace with no key rows |
| E-21 | Retire: namespace carries a tombstone | Step 1: internal/datastore/namespace_encryption_keys_rotation_integration_test.go; command level: Step 5 |
| E-22 | Retire: version at or above active | Step 1; command level: Step 5 |
| E-23 | Retire: version not found / no key rows | Step 1; command level: Step 5 |
| E-24 | Retire: FK-referenced version | Step 1 (real referencing table); command level: Step 5 |
| E-25 | Database unreachable | Step 2: TestRunOperator_DatabaseUnreachableSurfacedAsClientStartError |
Security considerations
| # | Concern | Tests |
|---|---|---|
| S-1 | AppSec + internal security review gate | Process gate on the MR, not testable in code |
| S-2 | Key material memory handling (Zeroize, copy-on-serve) |
S04-A internal/crypto suite; Step 4: TestBuildRootRotations_ZeroizesKeysAfterBuild |
| S-3 | No key material or credential values in logs; fixed log keys | This MR: AC-14 row tests plus TestReEncryptNamespace_LogFieldKeysAreStableConstants; Steps 2, 4: TestRunOperator_LogsCarryNoKeyMaterial, root-scope log capture |
| S-4 | Access control is the deployment boundary | Deployment property; not testable in this repository |
| S-5 | Backup retention window enforced by runbook | Step 5 (pre-flight age logging); Step 9 (runbook page) |
| S-6 | Commands run with the fleet's configuration (TTL parity) | TestRunOperator_RuntimeCarriesCacheTTL pins the guards read the loaded TTL; the mounted-config obligation itself is operational |
| S-7 | FIPS posture logged, not gated | Step 2: TestRunOperator_LogsFIPSVersionAtStartup |
Related to https://gitlab.com/gitlab-org/ops/artifact-registry/-/issues/496