test(storage): add gofail-driven upload-pipeline fault conformance cases

What this MR adds

The S06 storage upload-session edge-case scenarios as reusable conformance tests, driven through gofail failpoints so the same cases run against the in-memory stub now and the real PostgreSQL/object-storage implementation later — the executable contract both implementations are bound to.

All scenarios live in internal/storage/testsuites/session_faults.go and are implementation-agnostic: the suite never imports gofail. Each scenario takes a FaultArmer the implementation supplies — the stub arms the matching gofail seam (active only under the test:storage-failpoints job); Step 9's PostgreSQL impl will arm the equivalent driver/DB fault. No stub-specific tests are introduced — the previously stub-internal D2b and offset-divergence cases are now generic conformance cases.

Scenarios

Entry Scenario Seam / term
RunWriteStagingPartial A1a — partial Write is terminal writeStagingFail=1*off->1*return(3)
RunWriteStagingZero A1b — zero-byte Write stays resumable writeStagingFail=1*off->1*return(0)
RunCloseFlush G1a — Close buffer-flush failure is terminal closeFlushFail=1*off->1*return()
RunCommitFlush G1b — Commit buffer-flush failure is terminal commitFlushFail=return()
RunCommitFinalize D1a — finalize failure is terminal commitFinalizeFail=return()
RunCommitMove D1b — Move failure → finalize-only resume → retry commits commitMoveFail=1*return()
RunCommitTx D2a — DB-tx failure → DB-finalize-only resume → retry commits commitTxFail=1*return()
RunCommitTxUnrecoverable D2b — DB-tx failure + blob removed → unrecoverable commitTxFailRemoveBlob=1*return()
RunResumeOffsetDivergence resume offset-divergence guard rejects a backend-ahead-of-DB offset resumeOffsetDivergence=return

New stub seams

Two seams back the cases otherwise unreachable through the public BlobStore API, which is what lets D2b and offset-divergence be generic conformance cases instead of stub-internal tests:

  • commitTxFailRemoveBlob (commitDBTx) — removes the committed blob from destPath, then fails the transaction (D2b).
  • resumeOffsetDivergence (resumeRecover) — advances the observed backend offset past the persisted size_bytes.

Isolation

gofail state is process-global, so TestStub_SessionFaults is sequential and its subtests run one at a time. Go runs all non-parallel tests to completion before the parallel conformance suite resumes, so no armed seam leaks into a concurrent Write/Commit. The armSeam helper fails the test on a malformed term (rather than skipping), so a typo can't silently disable a case.

Validation

  • Default (disabled) tree: go build, go vet, golangci-lint 0 issues, go test -race — every fault case skips cleanly.
  • mise run test:storage-failpoints (enable → -race → disable): all 9 scenarios run and pass; the tree is restored byte-for-byte.
  • -count=2 under enable: no residual-seam flakiness.

Stacking

Stacked on prozlach/hoist-session-conformance-tests (which hoists the session conformance cases into the generic suite and carries the failpoint infrastructure + seams). This MR targets that branch; its diff is the scenario tests plus the two new seams.

🤖 Generated with Claude Code

Edited by Pawel Rozlach

Merge request reports

Loading
Loading