feat(cmd): namespace-scope rotate and sweep phases

Implements plan step 8 of docs/plans/2026-08-03-s04b-rotation-tooling.md: both phases of re-encrypt --scope=namespace, replacing the stub the operator subcommands shipped.

The rotate phase drives the namespace-key manager's rotation entry point behind the ActiveKeyAge guard: TTL-not-elapsed refuses at Warn with the age, the TTL, and an RFC3339 retry time ("if a second rotation is intended, retry after …"); shredded and no-key-row namespaces refuse without creating anything; success creates exactly one version and summarizes at Info with the created key_version.

The sweep phase re-checks the guard (refusing inside one TTL with the retry time), resolves the selection bound from the active key, and drives datastore.SweepCredentialTables over the composed registry: per-batch progress and per-table totals at Info (a failing table logs "table aborted" with its partial counts), abort or stale-active-key surfaces as exit 1, and the zero-registration sweep completes secret-read-free — the key manager is constructed only when at least one credential table is registered. The registry is composed empty in production; per-format registrations are #417 (closed)'s scope and inherit the composition's one-minute statement bound automatically (statementBoundCredentialTable wraps each table where the registry enters the sweep). The manager's own store calls are bounded the same way (statementBoundKeyStore).

No log line or error carries key material (pinned over the root key, KEKs, wrapped keys, and every wrapped DEK on both happy paths).

Stacking

Both parents (!1344 (merged) and !1412 (merged)) have merged, so this targets main and the diff is the step 8 change set.

Size

~2,470 added lines: ~530 implementation, the rest the unit matrix and the end-to-end integration suite the plan assigns to this step, plus the review-cycle hardening (deadline pins for both new statement-bound decorators, abort-summary and secret-decode pins, TTL boundary case) and the round-1 review fixes (goleak pins on both manager paths, the secret-read-free guard refusals, and the run's log-attribute contract).

Scope beyond the plan's step 8

Round 1 grew the step into internal/crypto, which the plan's step-8 entry now records. crypto.DefaultNamespaceKeyManager gained RotateKeyVersion, a rotation entry point that publishes nothing and returns only the version the store assigned; RotateKey is unchanged for callers that use the key they rotate to.

The rotate action needs a rotation to have happened and never uses the key, and two review findings were properties of the publishing primitive rather than of the call site: it hands package main a plaintext copy that package cannot wipe, and its publication step can refuse after the row is committed, reporting a live new version as a failed rotation. Both are fixed at the entry point instead of described at the call site.

E2E scenario catalogs

No scenario added or affected: operator-only rotation tooling; the catalogs cover format-client journeys.

Database review

No new migration or query-producing method — the phases drive the merged store operations and the step 7 seam.

Spec coverage

Spec: docs/specs/S04-b-encryption-key-rotation-tooling.md

Acceptance criteria

# Criterion Tests
AC-1 Root scope re-wraps every live row; re-run is zero rows Step 4 (on this base): TestReEncryptRootScope_RewrapsEveryLiveRow
AC-2 Single-key file still verifies Step 4: TestReEncryptRootScope_SingleKeyFileVerifies, TestReEncryptRootScope_SingleKeyFileVerifiesOffender
AC-3 Whole-live-set verification names offenders Step 4: TestReEncryptRootScope_StrandedURIFailsVerification, TestReEncryptRootScope_RewrapsEveryLiveRow
AC-4 Orientation refusal + startup URI logging Step 4: TestReEncryptRoot_OrientationRefusalNamesBothURIs, TestReEncryptRootScope_OrientationRefusalLeavesRowsUntouched
AC-5 Rotate creates exactly one version; shredded/no-rows/TTL refusals TestReEncryptNamespaceScope_RotateCreatesExactlyOneVersion, TestReEncryptNamespaceScope_RotateGuardRefusals, TestReEncryptNamespace_GuardRefusalsWarnAndExit1 (rotate rows)
AC-6 Sweep command-level clauses TTL refusal with retry time: TestReEncryptNamespace_SweepTTLRefusalNamesEarliestRetryTime, TestReEncryptNamespaceScope_SweepGuardOutcomes/before the ttl elapses; ordered run, selection bound, guarded write-back, skip counting, per-table counts: TestReEncryptNamespaceScope_SweepRunsRegisteredTablesInOrder; abort without later tables + duplicate TableName exit 1: TestReEncryptNamespaceScope_SweepAborts; zero registration: TestReEncryptNamespace_SweepZeroRegistrationSucceeds, TestReEncryptNamespaceScope_SweepGuardOutcomes/zero registered tables; shredded/no-rows: TestReEncryptNamespaceScope_SweepGuardOutcomes, TestReEncryptNamespace_GuardRefusalsWarnAndExit1 (sweep rows). Loop-behavior clauses (tripwire mechanics, duplicate rejection before any table): Step 7, internal/datastore/credential_sweep_test.go
AC-7 ErrStaleActiveKey aborts the sweep with exit 1 TestReEncryptNamespaceScope_SweepAborts/stale active key
AC-8 RetireKeyVersion contract and sentinel matrix Step 1: internal/datastore/namespace_encryption_keys_rotation_integration_test.go
AC-9 FK refusal mapped to ErrKeyStillReferenced Step 1: same file, via the testutil FK-table helper
AC-10 Retirement barrier concurrency Step 6 (test-only MR, not on this base)
AC-11 Usage-error matrix exits 1 before any database access Step 3: TestReEncryptCommand_UsageErrorsExitBeforeDatabase, retire_test.go
AC-12 Missing encryption block refused; database-only environment Steps 2-3: TestRunOperator_MissingEncryptionBlockRefusedBeforeDatabase, TestReEncryptCommand_MissingEncryptionBlockRefused
AC-13 Retire pre-flight logging Step 5 (not on this base)
AC-14 No key material in captured logs This MR for the namespace flows: TestReEncryptNamespaceScope_RotateCreatesExactlyOneVersion, TestReEncryptNamespaceScope_SweepRunsRegisteredTablesInOrder (both via assertLogsCarryNoKeyMaterial); other flows: Steps 2, 4, 5

Error cases

# Condition Tests
E-1 Usage errors Step 3: TestReEncryptCommand_UsageErrorsExitBeforeDatabase, retire_test.go
E-2 No encryption block Steps 2-3: TestRunOperator_MissingEncryptionBlockRefusedBeforeDatabase, TestReEncryptCommand_MissingEncryptionBlockRefused
E-3 Root-key secret missing or undecodable Steps 3-4: TestReEncryptRoot_SecretLoadFailureRefused, TestReEncryptRoot_MalformedSecretRefused, TestValidateRootKeySecret
E-4 Root scope: transient failure mid-run Step 4: TestReEncryptRootScope_TransientFailureResumes
E-5 Root scope: matched row fails to unwrap Step 4: TestReEncryptRootScope_UnwrapFailureAbortsAndPersists
E-6 Root scope: duplicate retained line skipped Step 4: TestReEncryptRootScope_DuplicateLineSkipped
E-7 Root scope: orientation mismatch Step 4: TestReEncryptRoot_OrientationRefusalNamesBothURIs, TestReEncryptRootScope_OrientationRefusalLeavesRowsUntouched
E-8 Interruption mid-run Step 2: TestRunOperator_SignalCancelsActionContext, TestRunOperator_SIGINTCancelsActionContext; resume: Step 4 (E-4 test); retirement atomicity: Step 1
E-9 Root scope: verification finds an offender Step 4: TestReEncryptRootScope_StrandedURIFailsVerification, TestReEncryptRootScope_SingleKeyFileVerifiesOffender
E-10 Rotate on shredded namespace TestReEncryptNamespaceScope_RotateGuardRefusals/shredded namespace, TestReEncryptNamespace_GuardRefusalsWarnAndExit1/rotate on shredded namespace
E-11 Rotate on namespace with no key rows TestReEncryptNamespaceScope_RotateGuardRefusals/no key rows, TestReEncryptNamespace_GuardRefusalsWarnAndExit1/rotate on namespace with no key rows
E-12 Rotate within one TTL TestReEncryptNamespaceScope_RotateGuardRefusals/within one ttl, TestReEncryptNamespaceScope_RotateCreatesExactlyOneVersion (second invocation), TestReEncryptNamespace_GuardRefusalsWarnAndExit1/rotate within one ttl of the active key
E-13 Sweep before one TTL, earliest retry time TestReEncryptNamespace_SweepTTLRefusalNamesEarliestRetryTime, TestReEncryptNamespaceScope_SweepGuardOutcomes/before the ttl elapses
E-14 Sweep row newer than resolved active key TestReEncryptNamespaceScope_SweepAborts/stale active key
E-15 Sweep with zero registered tables TestReEncryptNamespace_SweepZeroRegistrationSucceeds, TestReEncryptNamespaceScope_SweepGuardOutcomes/zero registered tables
E-16 Sweep table read/write fails mid-run Loop mechanics (abort, partial counts, later tables not attempted): Step 7, internal/datastore/credential_sweep_test.go; command-level error-to-exit-1 surfacing: TestReEncryptNamespaceScope_SweepAborts/stale active key
E-17 Two registered tables share a TableName Loop rejection before any table: Step 7; command-level exit 1: TestReEncryptNamespaceScope_SweepAborts/duplicate table name
E-18 Over-long batch or non-advancing cursor Tripwire mechanics: Step 7, internal/datastore/credential_sweep_test.go; the command surfaces any loop error as exit 1 (path pinned by TestReEncryptNamespaceScope_SweepAborts/stale active key)
E-19 Sweep on shredded namespace TestReEncryptNamespaceScope_SweepGuardOutcomes/shredded namespace, TestReEncryptNamespace_GuardRefusalsWarnAndExit1/sweep on shredded namespace
E-20 Sweep on namespace with no key rows (no auto-create) TestReEncryptNamespaceScope_SweepGuardOutcomes/no key rows, TestReEncryptNamespace_GuardRefusalsWarnAndExit1/sweep on namespace with no key rows
E-21 Retire: namespace carries a tombstone Step 1: internal/datastore/namespace_encryption_keys_rotation_integration_test.go; command level: Step 5
E-22 Retire: version at or above active Step 1; command level: Step 5
E-23 Retire: version not found / no key rows Step 1; command level: Step 5
E-24 Retire: FK-referenced version Step 1 (real referencing table); command level: Step 5
E-25 Database unreachable Step 2: TestRunOperator_DatabaseUnreachableSurfacedAsClientStartError

Security considerations

# Concern Tests
S-1 AppSec + internal security review gate Process gate on the MR, not testable in code
S-2 Key material memory handling (Zeroize, copy-on-serve) S04-A internal/crypto suite; Step 4: TestBuildRootRotations_ZeroizesKeysAfterBuild
S-3 No key material or credential values in logs; fixed log keys This MR: AC-14 row tests plus TestReEncryptNamespace_LogFieldKeysAreStableConstants; Steps 2, 4: TestRunOperator_LogsCarryNoKeyMaterial, root-scope log capture
S-4 Access control is the deployment boundary Deployment property; not testable in this repository
S-5 Backup retention window enforced by runbook Step 5 (pre-flight age logging); Step 9 (runbook page)
S-6 Commands run with the fleet's configuration (TTL parity) TestRunOperator_RuntimeCarriesCacheTTL pins the guards read the loaded TTL; the mounted-config obligation itself is operational
S-7 FIPS posture logged, not gated Step 2: TestRunOperator_LogsFIPSVersionAtStartup


Related to https://gitlab.com/gitlab-org/ops/artifact-registry/-/issues/496
Edited by Suleimi Ahmed

Merge request reports

Loading
Loading