test(storage): add gofail-driven upload-pipeline fault conformance cases
What this MR adds
The S06 storage upload-session edge-case scenarios as reusable conformance tests, driven through gofail failpoints so the same cases run against the in-memory stub now and the real PostgreSQL/object-storage implementation later — the executable contract both implementations are bound to.
All scenarios live in internal/storage/testsuites/session_faults.go and are implementation-agnostic: the suite never imports gofail. Each scenario takes a FaultArmer the implementation supplies — the stub arms the matching gofail seam (active only under the test:storage-failpoints job); Step 9's PostgreSQL impl will arm the equivalent driver/DB fault. No stub-specific tests are introduced — the previously stub-internal D2b and offset-divergence cases are now generic conformance cases.
Scenarios
| Entry | Scenario | Seam / term |
|---|---|---|
RunWriteStagingPartial |
A1a — partial Write is terminal |
writeStagingFail=1*off->1*return(3) |
RunWriteStagingZero |
A1b — zero-byte Write stays resumable |
writeStagingFail=1*off->1*return(0) |
RunCloseFlush |
G1a — Close buffer-flush failure is terminal |
closeFlushFail=1*off->1*return() |
RunCommitFlush |
G1b — Commit buffer-flush failure is terminal |
commitFlushFail=return() |
RunCommitFinalize |
D1a — finalize failure is terminal | commitFinalizeFail=return() |
RunCommitMove |
D1b — Move failure → finalize-only resume → retry commits |
commitMoveFail=1*return() |
RunCommitTx |
D2a — DB-tx failure → DB-finalize-only resume → retry commits | commitTxFail=1*return() |
RunCommitTxUnrecoverable |
D2b — DB-tx failure + blob removed → unrecoverable | commitTxFailRemoveBlob=1*return() |
RunResumeOffsetDivergence |
resume offset-divergence guard rejects a backend-ahead-of-DB offset | resumeOffsetDivergence=return |
New stub seams
Two seams back the cases otherwise unreachable through the public BlobStore API, which is what lets D2b and offset-divergence be generic conformance cases instead of stub-internal tests:
commitTxFailRemoveBlob(commitDBTx) — removes the committed blob from destPath, then fails the transaction (D2b).resumeOffsetDivergence(resumeRecover) — advances the observed backend offset past the persistedsize_bytes.
Isolation
gofail state is process-global, so TestStub_SessionFaults is sequential and its subtests run one at a time. Go runs all non-parallel tests to completion before the parallel conformance suite resumes, so no armed seam leaks into a concurrent Write/Commit. The armSeam helper fails the test on a malformed term (rather than skipping), so a typo can't silently disable a case.
Validation
- Default (disabled) tree:
go build,go vet,golangci-lint0 issues,go test -race— every fault case skips cleanly. mise run test:storage-failpoints(enable →-race→ disable): all 9 scenarios run and pass; the tree is restored byte-for-byte.-count=2under enable: no residual-seam flakiness.
Stacking
Stacked on prozlach/hoist-session-conformance-tests (which hoists the session conformance cases into the generic suite and carries the failpoint infrastructure + seams). This MR targets that branch; its diff is the scenario tests plus the two new seams.